hao-ai-lab-FastVideo/docs/inference/deployment_spark.md

MiniMax H3 on NVIDIA DGX Spark (GB10)

This guide covers running MiniMax H3 / FastH3 inference on DGX Spark clusters (GB10, 128 GB unified memory per node) and the encoder split mode that makes 720p fit without host offload.

Why a dedicated encoder node group

Measured resident sizes on GB10 (NVFP4 checkpoint, lazy load):

component	size
text_encoder (Qwen3-VL)	47.98 GiB
transformer (DiT)	65.53 GiB
video VAE	9.74 GiB
audio VAE	0.58 GiB

All three heavy groups resident at once is ~124 GiB, above the ~115 GiB usable on a 128 GB GB10, so 720p (768x1344x124) OOMs on a single node no matter what offload knobs are set. Sequential/lazy loading already avoids encoding and denoising at the same time; the encoder split goes one step further and runs them on different nodes:

• the first N nodes (world ranks 0..N-1) load only the Qwen3-VL conditioner, run input_preparation + conditioning, and NCCL-broadcast prompt_embeds to everyone over the world group;
• the remaining nodes load only DiT + VAEs (never the text encoder) and form the single sequence-parallel group, sp_size = num_gpus - encoder_workers.

A denoise node then peaks around ~76 GiB plus activations, which leaves plenty of headroom for 720p.

Enabling

The feature is off by default and only valid with the Ray backend.

export FASTVIDEO_H3_ENCODER_SPLIT=1   # or FastVideoArgs.h3_encoder_split / --h3-encoder-split
export FASTVIDEO_H3_ENCODER_NODES=1   # nodes dedicated to the encoder; default 1

fastvideo generate exposes only --config, so pass the same two settings as dotted overrides into the experimental catch-all: --generator.pipeline.experimental.h3_encoder_split true and --generator.pipeline.experimental.h3_encoder_nodes 1.

Cluster-side prerequisites (all nodes): the FastVideo venv, the model path, and per-node fabric exports (NCCL_SOCKET_IFNAME / GLOO_SOCKET_IFNAME / NCCL_IB_HCA set before ray start, since the executor deliberately does not overwrite these per-node values). Start one Ray head and join the workers as usual; pass --num-gpus <total nodes> and --execution-backend ray.

Choosing the encoder-node count

The denoise group runs sequence parallelism, so its size must divide the DiT attention head count (56 in the released FastH3 checkpoints). Valid encoder node counts are therefore constrained: with 8 nodes only 1, 4, 6, 7 are legal (SP = 7, 4, 2, 1). This is validated before any weights load, with the legal values listed in the error.

One request packs a single prompt presentation, so world rank 0 is the only encoder that computes; extra encoder ranks mirror the broadcast collectives and stay idle (reserved capacity, not a speedup for single-prompt requests).

Verified run (8x GB10, NVFP4 checkpoint, TAEH3 preview decode)

Encoder node = rank 0, denoise SP = 7, 5 DMD steps:

geometry	result	wall
256x448x124	ok	29.2 s cold
768x1344x124 (720p)	ok	51.6 s cold / 36.9 s steady

Without the split, the same 720p request reaches ~104 GiB while loading the DiT after conditioning and OOMs inside safetensors persistent_load.

Troubleshooting

• attention heads (56) must be divisible by sequence parallel size: the encoder-node count left an illegal denoise-group size; use one of the values in the message.
• Drivers hang with no stage events and node memory stuck mid-load: check the cluster is actually free (ray status should show 0.0/8.0 GPU and no actors). A leftover job holding the placement group starves the new run, and a full VAE decode of 720p can die at the NVRM level (dmesg | grep NVRM) with no Python traceback - prefer --video-decode-backend taeh3 at 720p on 128 GB nodes.
• The prompt_embeds broadcast travels over the world NCCL group, not the Ray object store (whose 4 GB cap spills large tensors to disk).