A set of custom nodes designed for experiments with video diffusion models. EXPERIMENTAL, and I mean it. Constantly updating, changing, adding and removing, just for sake of making something work. You have been warned.

FunPack CLIP Loader

This node is designed specifically for FramePack/HunyuanVideo, aiming to replace text encoder module with LLama-3 Instruct model. Not entirely, just text encoder. Vision module stays the same, so you'll need original llava-llama-3 as well.

Inputs:

clip_model_name - your CLIP-L model that you usually use with Hunyuan/FramePack (e.g. clip-vit-large-patch14);
text_encoder_model_name - your instruct (or any other LLM?) model. Expects .safetensors file, if "instruct_from_pretrained" is on - ignores this;
llm_vision_model_name - your llava-llama-3 model you usually use with Hunyuan/FramePack (e.g. llava-llama-3-8b-v1_1). Also it's possible to load any other LLM, with or without vision capabilities (I guess);
type - select "hunyuan_video", left for compatibility;
encoder_pretrained_path - Provide a HuggingFace path for config and tokenizer for your encoder model (or for weights as well, if encoder_from_pretrained=True);
vision_pretrained_path - Provide a HuggingFace path for LLM+vision model (only used if vision_from_pretrained=True);
encoder_from_pretrained - if enabled, loads encoder model weights from encoder_pretrained_path as well, ignoring local "text_encoder_model_name";
vision_from_pretrained - if enabled, loads LLM+vision model weights from vision_pretrained_path, ignoring local "llm_vision_model_name";
load_te - if enabled, loads your custom text encoder model. If disabled, uses only vision one (e.g. llava-llama-3-8b-v1_1);
system_prompt - your system prompt that Instruct model is going to be using.
top_p, top_k, temperature - these are parameters for generating an "assistant prompt". It goes here:

def tokenize(self, text): assistant_reply = self.generate(text) messages = [ {"role": "system", "content": self.system_prompt}, {"role": "user", "content": text}, {"role": "assistant", "content": assistant_reply} ] return tokenizer.apply_chat_template(messages, add_generation_prompt=False, return_tensors="pt").to("cuda")

Technically speaking, it's possible to load just any model as text encoder.

Outputs: Just CLIP. Pass it through your nodes like you will do with regular DualCLIPLoader.

FunPack img2latent Interpolation

Inputs:

images - a batch of images e.g. from Load Image, Load Video (Upload), Load Video (Path) et cetera.
frame_count - same as frame count in HunyuanVideoEmptyLatent or similar. Actually, works better with WAN.

Outputs:

img_batch_for_encode - latent output with interpolated image, resize if needed and put it into VAE Encode, pass as latent_image into your sampler;
img_for_start_images - takes exactly the last image from the batch. You can pass it to your CLIP Vision and encoders as start_image. Or end_image. Who am I to tell you.