✨ ComfyUI-PHRenderFormerWrapper ✨

Author: paulh4x

This repository contains a set of custom nodes for ComfyUI that provide a wrapper for Microsoft's RenderFormer model. It allows you to render complex 3D scenes (see Limits) with physically-based materials and global illumination directly within the ComfyUI interface.

Warning

WORK IN PROGRESS & HELLO WORLD: This project is my first "hello world" in contributing code of any kind and is currently under active development. It should be considered experimental. The model is in a very early stage and may be further developed later. I do not take responsibility if this breaks anything and do not plan to provide official support for it.

---

💡 Usage

This node pack follows a standard 3D rendering pipeline. The basic workflow, as seen in the ph_renderformer_basic_01.json example, can be broken down into the following steps:

graph TD
    subgraph "Step 1: Load Model"
        A["RenderFormer Model Loader"]
    end

    subgraph "Step 2: Define Scene Elements"
        B["RenderFormer Mesh Loader"] --> C["RenderFormer Random Colors"]
        D["RenderFormer Camera"]
        E["RenderFormer Lighting"]
    end

    subgraph "Step 3: Build Scene"
        C --> F["RenderFormer Scene Builder"]
        D --> F
        E --> F
    end

    subgraph "Step 4: Render (Sample)"
        A --> G["RenderFormer Sampler"]
        F --> G
    end

    subgraph "Step 5: Save Output"
        G --> H["Save Image"]
    end

    style A fill:#FDC501,stroke:#111417,stroke-width:2px,color:#111417
    style B fill:#FDC501,stroke:#111417,stroke-width:2px,color:#111417
    style C fill:#FDC501,stroke:#111417,stroke-width:2px,color:#111417
    style D fill:#FDC501,stroke:#111417,stroke-width:2px,color:#111417
    style E fill:#FDC501,stroke:#111417,stroke-width:2px,color:#111417
    style F fill:#FDC501,stroke:#111417,stroke-width:2px,color:#111417
    style G fill:#FDC501,stroke:#111417,stroke-width:2px,color:#111417
    style H fill:#a0a0a0,stroke:#111417,stroke-width:2px,color:#111417

1. Load Model: Start by loading the RenderFormer Model Loader. This node downloads the specified model from Hugging Face and prepares it for the pipeline.
2. Define Scene Elements:
   • Mesh: Use the RenderFormer Mesh Loader to load your 3D model (e.g., an .obj file). Here, you can also set its transformation (translation, rotation, scale) and define its core material properties. In the example, the output is passed to RenderFormer Random Colors for a creative effect.
   • Camera: Add a RenderFormer Camera to define the viewpoint, target, and field of view.
   • Lighting: Use the RenderFormer Lighting node to create one or more light sources. You can combine multiple lights with the RenderFormer Lighting Combine node.
3. Build Scene: Connect the MESH, CAMERA, and LIGHTING outputs into the RenderFormer Scene Builder. This node collects all the components and assembles the final scene data that the model will render. You can also choose to add a default background plane and walls here.
4. Render (Sample): The RenderFormer Sampler takes the MODEL and the SCENE as input. It performs the actual rendering operation and outputs the final image.
5. Save Output: Connect the IMAGE output from the sampler to a Save Image node to save the result.

For video, the workflow is similar. You would use a RenderFormer Camera Target to create a CAMERA_SEQUENCE, which then goes into the RenderFormer Scene Builder. The RenderFormer Sampler will automatically detect the sequence and output an image batch (IMAGES) ready for a video-saving node.

---

🚀 Advanced Usage: Building Complex Scenes & Animations

For more complex projects, you can combine multiple meshes and create camera animations. The ph_renderformer_advanced_01.json workflow demonstrates this by building a scene with several objects and animating the camera.

graph TD
    subgraph "Step 1: Load Model"
        A["RenderFormer Model Loader"]
    end

    subgraph "Step 2: Define Scene Elements"
        subgraph "Compose Meshes"
            B1["Mesh 1 (Main Object)"] --> C["Mesh Combine"]
            B2["Mesh 2 (Fox)"] --> C
            B3["Mesh 3 (Spheres)"] --> C
            B4["Mesh 4 (Background Walls)"] --> C
        end

        subgraph "Animate Camera"
            D1["RenderFormer Camera (Start)"] --> D2["RenderFormer Camera Target (End)"]
        end
        
        E["RenderFormer Lighting"]
    end

    subgraph "Step 3: Build Scene Sequence"
        C --> F["RenderFormer Scene Builder"]
        D2 -- CAMERA_SEQUENCE --> F
        E --> F
    end

    subgraph "Step 4: Render (Sample)"
        A --> G["RenderFormer Sampler"]
        F -- SCENE_SEQUENCE --> G
    end

    subgraph "Step 5: Create & Save Video"
        G -- IMAGES --> H["Create Video"]
        H --> I["Save Video"]
    end

    style A fill:#FDC501,stroke:#111417,stroke-width:2px,color:#111417
    style B1 fill:#FDC501,stroke:#111417,stroke-width:2px,color:#111417
    style B2 fill:#FDC501,stroke:#111417,stroke-width:2px,color:#111417
    style B3 fill:#FDC501,stroke:#111417,stroke-width:2px,color:#111417
    style B4 fill:#FDC501,stroke:#111417,stroke-width:2px,color:#111417
    style C fill:#FDC501,stroke:#111417,stroke-width:2px,color:#111417
    style D1 fill:#FDC501,stroke:#111417,stroke-width:2px,color:#111417
    style D2 fill:#FDC501,stroke:#111417,stroke-width:2px,color:#111417
    style E fill:#FDC501,stroke:#111417,stroke-width:2px,color:#111417
    style F fill:#FDC501,stroke:#111417,stroke-width:2px,color:#111417
    style G fill:#FDC501,stroke:#111417,stroke-width:2px,color:#111417
    style H fill:#a0a0a0,stroke:#111417,stroke-width:2px,color:#111417
    style I fill:#a0a0a0,stroke:#111417,stroke-width:2px,color:#111417

1. Compose Your Scene: Instead of using one large model, you can load multiple RenderFormer Mesh Loader nodes for different parts of your scene (e.g., characters, props, background elements).
2. Combine Meshes: Use one or more RenderFormer Mesh Combine nodes to merge all your individual meshes into a single MESH output. This creates a clean, organized workflow and makes it easy to manage complex scenes.
3. Animate the Camera:
   • Define the starting view with a RenderFormer Camera node.
   • Connect its CAMERA output to the start_camera input of a RenderFormer Camera Target node.
   • In the Camera Target node, define the end position, look-at point, and FOV for your animation, along with the total number of frames. This generates a CAMERA_SEQUENCE.
4. Build the Scene Sequence: Connect your combined MESH, the CAMERA_SEQUENCE, and your LIGHTING to the RenderFormer Scene Builder. It will automatically create a sequence of scenes, one for each frame of the animation.
5. Render and Save:
   • The RenderFormer Sampler will receive the SCENE_SEQUENCE and render all frames, outputting them as an IMAGES batch.
   • Connect this batch to a Create Video node (or another video-handling node) and then to a Save Video node to get your final animation file.

---

🚀 Features

• 🎨 End-to-End Rendering: Load 3D models, define materials, set up cameras, and render—all within ComfyUI.
• ⚙️ Modular Node-Based Workflow: Each step of the rendering pipeline is a separate node, allowing for flexible and complex setups.
• 🎥 Animation & Video: Create camera animations by interpolating between keyframes. The nodes output image batches compatible with ComfyUI's native video-saving nodes.
• 🔧 Advanced Mesh Processing: Includes nodes for loading, combining, remeshing, and applying simple color randomization to your 3D assets.
• 💡 Lighting and Material Control: Easily add and combine multiple light sources and control PBR material properties like diffuse, specular, roughness, and emission.
• ↔️ Full Transformation Control: Apply translation, rotation, and scaling to objects within the scene.

---

🧱 Model Limits

• Meshes: RenderFormer can handle scenes of up to 8192 polygons.
• Lights: Up to 8 lightsources can be combined for lighting, emission color is for now limited to rgb 255, 255, 255 (white).
• Resolution: Best tested resolutions for now are between 512 x 512 to 1024 x 1024 pixels. The model can produce resolutions up to 2048 x 2048 pixel, quality of outputs decrease with higher resolutions (see comparison img).
• Animations: Due to slightly varying precision in each frame rendered, camera animations for now contain some flickering, especially with high reflective materials.

📝 Progress & To-Do

This project is under active development. Here is a summary of the progress so far and the features planned for the future.

✅ Done

• Core Rendering Pipeline: All essential nodes for building and rendering a static 3D scene are implemented.
• In-Memory Processing: Partitial scene preparation pipeline is handled in-memory to maximize speed.
• Video Rendering: A complete, dedicated workflow for video rendering is in place.
• Advanced Utilities: Nodes for mesh combination, remeshing, and JSON-based scene loading are functional.
• Custom UI Elements: The nodes feature custom colors for better visual organization, and progress bars are implemented for long-running operations.
• Bug Fixes: Addressed various bugs related to file handling, data types, and temporary file management.
• Proper RGB Values for diffuse Color: Colorpicker support for white-values for more variety of colors including.

📋 To Do

• Randomize Color Fix: Fix RandomizeColors node, will produce the expected visual output.
• ⚡ In-Memory Processing: Fix entire scene preparation pipeline, from loading meshes to generating the final HDF5 data for the model, should be handled in-memory to maximize speed and avoid disk I/O.
• Camera Animation: Fix flickering in animations especially for objects with high glossy materials.
• Mesh Format Support: Add support for loading .glb and .fbx files, including their materials and textures.
• Target Animation Nodes: Implement animation capabilities for MESH and LIGHT properties, not just the camera.
• Camera Adoption: Integrate with the Load 3D core node to adopt its camera transformations.
• Material Presets: Create a system for saving and loading material presets.
• Public Release: Prepare for a more stable, public release with better documentation and examples.

---

🛠️ Installation

Prerequisites

• Git: Required for cloning the repository and installing certain dependencies.
• Python: A Python version compatible with ComfyUI and PyTorch 2.0+.
• PyTorch: A compatible version of PyTorch must be installed for your hardware (NVIDIA CUDA or Apple Metal). This wrapper is tested with PyTorch 2.7.1 and CUDA 12.6.
• ComfyUI: A working installation of ComfyUI.

Installation Steps

1. Navigate to your ComfyUI custom_nodes directory:

   cd ComfyUI/custom_nodes/
   

2. Clone this repository:

   git clone https://github.com/your-github-username/ComfyUI_PHRenderFormerWrapper.git
   

3. Navigate into the newly cloned directory:

   cd ComfyUI_PHRenderFormerWrapper
   

4. Clone the official Microsoft RenderFormer repository into this directory. It must be named renderformer:

   git clone https://github.com/microsoft/renderformer.git renderformer
   

5. Install the required Python packages for this wrapper:

   pip install -r requirements.txt
   

Important

For owners of a NVIDIA 50xx series GPU, a dedicated requirements file is available. Please use the following command instead:

pip install -r requirements_50xx.txt

6. RenderFormer requires an additional plugin for handling HDR image formats. Run the following command to download it:

   python -c "import imageio; imageio.plugins.freeimage.download()"
   

7. Restart ComfyUI.

Note

For NVIDIA GPU users, the underlying RenderFormer library can optionally use Flash Attention for better performance. If you have it installed in your PyTorch environment, it should be used automatically.

---

📦 Nodes

This wrapper provides a comprehensive set of nodes to build 3D scenes.

Core Pipeline

• RenderFormer Model Loader: Loads a specified RenderFormer model from Hugging Face or a local path.
• RenderFormer Scene Builder: Assembles a scene from meshes, lighting, and camera inputs. It can handle both single camera inputs for static images and camera sequences for creating animations.
• RenderFormer Sampler: Executes the RenderFormer pipeline on a scene or a sequence of scenes. It intelligently handles both single images and image batches for video, producing the final rendered output.

Scene Components

• RenderFormer Mesh Loader: Loads a 3D mesh file (e.g., .obj, .glb). This node is also used to define an object's material properties and transformations.
• RenderFormer Camera: Defines the camera's position, look-at target, and field of view (FOV).
• RenderFormer Lighting: Creates a configurable emissive light source.

Animation

• RenderFormer Camera Target: Creates a camera animation sequence by interpolating between a start and end camera state (position, look-at, FOV).

Utilities

• RenderFormer Mesh Combine: Combines multiple MESH outputs into a single object list.
• RenderFormer Lighting Combine: Combines multiple light sources into a single list for the scene builder.
• RenderFormer Remesh: Simplifies the geometry of a mesh to a target face count using pymeshlab.
• RenderFormer Random Colors: Applies random vertex colors to a mesh for creative effects or debugging.

Advanced & Experimental

• RenderFormer From JSON: Loads a scene from a JSON definition, allowing for more complex and customized setups based on the original RenderFormer format.
• RenderFormer Example Scene: A test node to quickly load one of the official RenderFormer example scenes.

---

📜 Version History

Version 0.2.5 - Node Consolidation and Workflow Simplification

• Workflow: Merged the RenderFormerVideoSamplerBatched into the main RenderFormerGenerator (now RenderFormer Sampler). The sampler now intelligently handles both single images and video sequences, with an optional IMAGES output.
• Workflow: Merged the RenderFormerVideoSceneBuilder into the main RenderFormerSceneBuilder. The node now accepts both a single CAMERA and a CAMERA_SEQUENCE, with optional SCENE and SCENE_SEQUENCE outputs.
• Fix: Resolved a critical AttributeError in the RenderFormerSceneBuilder caused by a missing helper method after the node merge.
• Fix: Removed hardcoded material properties for background meshes in the RenderFormerLoadMesh node, ensuring that materials are always controlled by the node's inputs.
• UI/UX: Added a colored startup message with an ASCII logo to provide clear feedback on whether the nodes loaded successfully.
• UI/UX: Fixed a SyntaxWarning related to escape sequences in the startup message.

Version 0.2.21 Beta Test - UI/UX and Workflow Enhancements

• Examples: added example workflows for ComfyUI.
• UI/UX: All nodes now have custom colors for better visual organization in the graph. The header is yellow (#FDC501) and the body is anthracite (#111417).
• UI/UX: Added progress bars to all long-running nodes (SceneBuilder, VideoSceneBuilder, Sampler, VideoSampler, Remesh) to provide real-time feedback.
• Workflow: The Remesh node now processes all meshes in a list, not just the first one.
• Workflow: The SceneBuilder nodes will no longer add the default background if a background mesh has already been loaded manually, preventing duplicates.
• Input Precision: The emissive_strength input on the Lighting node now has a smaller step value for finer control.

Version 0.1.9 - Video Rendering and Stability

• Feature: Added a complete, dedicated workflow for video rendering (Camera Target, Video Scene Builder, Video Sampler).
• Fix: Resolved a dacite.exceptions.MissingValueError in the video scene builder.
• Fix: Addressed TypeError and FileNotFoundError bugs related to temporary file handling in the video pipeline.
• Change: Separated single-image and video workflows for improved stability and clarity.

Version 0.1.5 - Refinements and Fixes

• Feature: The LoadMesh node now has a RENDERFORMER_MATERIAL output, allowing material properties to be shared.
• Feature: Added default transformations for specific background meshes from the official examples.
• Change: Increased input precision for rotation and scale transformations.

Version 0.1.0 - Initial Development

• Feature: Established the core rendering pipeline and all essential nodes for building and rendering a static 3D scene.
• Feature: Implemented in-memory scene processing to patch trimesh and h5py, avoiding slow disk I/O.
• Feature: Added advanced utility nodes like RemeshMesh, RandomizeColors, and FromJSON.
• UI/UX: Integrated a file uploader and a custom canvas-based color picker for a better user experience.

---

🌱 My Journey

This project represents my first steps into coding and open-source contribution. It was born out of a desire to learn and create. What started as a simple experiment has been a journey of discovery, and I'm excited to see where it goes. I would love to develop this project further with the help of the community and welcome any contributions or feedback.

🙏 Acknowledgements

This project would not be possible without the foundational work of others.

• RenderFormer This project is a wrapper for the incredible RenderFormer model. All credit for the underlying rendering technology goes to the original authors: Chong Zeng, Yue Dong, Pieter Peers, Hongzhi Wu, and Xin Tong.

  • Official RenderFormer Project Page
  • Official RenderFormer GitHub

• comfyui-hunyuan3dwrapper Special thanks to kijai for their work on the comfyui-hunyan3dwrapper, which served as an invaluable reference and starting point for this project during its development in vibecoding.

  • ComfyUI-Hunyuan3DWrapper GitHub

• ComfyUI_Fill-Example-Nodes A huge thank you to filliptm for creating the ComfyUI_Fill-Example-Nodes repository. The advanced color picker in this project was implemented by adapting the excellent example code provided in that repo. It was an essential learning resource for understanding how to build custom UI elements in ComfyUI.

  • ComfyUI_Fill-Example-Nodes GitHub