SolitaryThinker
0467edbd07
Completes the bucket-C porting backlog. Each arch is a self-contained recipe
package (card-declared torch adapter via ComponentSpec.adapter + a new/forked
loop + program, NO edit to the shared torch_backend dispatch), following the
cosmos2 reference pattern. One _BUCKET_C table in v2/registry.py drives both the
explicit HF-id registry (PRIMARY) and the select_by_architecture fallback.
Ported (CPU-verified: import + card/program build + registry resolve + denoise
loop runs end-to-end on the CPU toy backend; GPU load/run is BRINGUP):
* cosmos25 — Cosmos-Predict2.5 (flow-match, per-frame plain-sigma timestep;
reuse FLOW_MATCH_STEP; Reason1/Qwen2.5-VL encoder adapter)
* hunyuan_video — HunyuanVideo (reuses WanDenoiseLoop; dual LLaMA+CLIP encoders;
Hunyuan VAE scaling_factor) + FastHunyuan variant
* hunyuan_video15 — HunyuanVideo 1.5 (480p/720p cards)
* longcat — LongCat-Video T2V/I2V/VC
* sd35 — SD3.5 MMDiT (flow-match, image; triple-encoder joint embed +
pooled_projections)
* gen3c — GEN3C (EDM; 82ch pose-buffer DiT; camera/depth -> BRINGUP)
* kandinsky5 — Kandinsky 5.0 T2V Lite
* flux2 — FLUX.2 dev/klein (MMDiT, image; gated weights -> BRINGUP)
* stable_audio — Stable Audio Open (audio modality)
* hunyuangamecraft, hyworld, lingbotworld, matrixgame2, matrixgame3 — interactive
world models; t2v/degenerate path CPU-verified, action/camera/
memory conditioning is BRINGUP (needs request-API extension)
Adapters declared via ComponentSpec.adapter (the ac29750b enabler) so each port
adds only NEW files (recipe package + per-arch torch_<arch>.py + optional facade
stub) — zero shared-file edits. Registry resolves all 31 bucket-C HF ids by exact
id + 14 architecture fallbacks; no regression (cosmos2/wan/ltx2 unchanged).
v2 mini green (211 passed, 2 skipped); pre-commit clean (per-file/registry).
64 lines
4.0 KiB
Python
64 lines
4.0 KiB
Python
"""LongCat-Video torch adapter (GPU backend) — declared on the card via ``ComponentSpec.adapter`` so
|
|
the LongCat recipe is self-contained (no edit to the shared ``_make_dit`` dispatch in ``torch_backend.py``).
|
|
Imported lazily by ``_explicit_adapter`` only on a GPU box.
|
|
|
|
* ``LongCatDiT`` — the flow-match velocity predictor. Mirrors ``WanDiT`` (``timestep = sigma*1000``,
|
|
scalar ``[B]`` timestep that the DiT internally expands ``[B] -> [B, T_latent]`` for the per-latent-frame
|
|
AdaLN time embedding) but with the **critical LongCat sign convention**: the fastvideo
|
|
``LongCatDenoisingStage`` does ``noise_pred = -noise_pred`` *before* the flow-match scheduler step
|
|
(longcat_denoising.py line 152). The v2 loop integrates ``x + (sigma_next-sigma)*velocity`` with the
|
|
velocity this adapter returns, so to reproduce that step EXACTLY this adapter returns ``-velocity``.
|
|
Forgetting the negation silently produces diverging samples (spec blocker #1).
|
|
|
|
LongCat reuses the existing ``WanVAE`` torch adapter unchanged (same ``AutoencoderKLWan`` mean/std
|
|
normalization) and the existing ``T5Encoder`` (UMT5, zero-padded to max_length=512 for the CFG concat
|
|
uniform-seq contract). Both are wired on the card by ``load_id`` alone — no adapter override needed.
|
|
|
|
The DiT runs its AdaLN modulation / residual gating + final projection under fp32 and casts the output to
|
|
float32 (longcat.py line 1121); we keep the module at its native bf16 dtype (``_native_dtype`` in
|
|
``_make_dit``, matching the fastvideo stage's hardcoded bf16 target) and let ``_n`` marshal the fp32 output
|
|
back to numpy (spec blocker #6).
|
|
|
|
BRINGUP: written-not-run on CPU (no torch/weights here). GPU-verify against a real
|
|
``FastVideo/LongCat-Video-T2V-Diffusers`` checkpoint — confirm the bf16 dtype, the scalar-timestep
|
|
expansion, and ``encoder_attention_mask=None`` (the CaptionEmbedder simply skips the zeroing branch for
|
|
T2V, which is faithful since the v2 ``T5Encoder`` already zero-pads the embedding rows past real tokens).
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import torch
|
|
|
|
from v2.platform.backends.torch_backend import NUM_TRAIN_TIMESTEPS, TorchComponent
|
|
|
|
|
|
class LongCatDiT(TorchComponent):
|
|
"""``dit(latent[C,T,H,W], text_embed[seq,4096], sigma) -> -velocity[C,T,H,W]``.
|
|
|
|
Real forward (``fastvideo/models/dits/longcat.py``):
|
|
``forward(hidden_states[B,C=16,T,H,W], encoder_hidden_states[B,N_text,4096], timestep[B]|[B,T],
|
|
encoder_attention_mask=None, ...) -> velocity[B,C_out=16,T,H,W]`` (cast to float32).
|
|
|
|
The loop hands the raw ``sigma`` (1->0); this adapter forms ``timestep = sigma*1000`` (the Wan/diffusers
|
|
convention LongCat shares) as a scalar ``[B]`` tensor — the DiT's ``forward`` expands ``[B] -> [B, T_latent]``
|
|
internally for the AdaLN time embedding (longcat.py line 1075). It returns the **negated** velocity to
|
|
fold in the fastvideo stage's ``noise_pred = -noise_pred`` (spec blocker #1)."""
|
|
|
|
@torch.no_grad()
|
|
def __call__(self, latent, text_embed, sigma, context=None, *, cond=None):
|
|
hs = self._t(latent) # [1, C=16, T, H, W]
|
|
ehs = self._t(text_embed) # [1, N_text, 4096]
|
|
b = hs.shape[0]
|
|
# timestep = sigma*1000 (BRINGUP risk B / spec timestep_convention). Scalar [B]; the DiT expands
|
|
# [B] -> [B, T_latent] internally for the per-latent-frame AdaLN time embedding. (I2V/VC would build
|
|
# an explicit [B, T_latent] with cond frames zeroed — deferred, T2V uses the scalar path.)
|
|
ts = float(sigma) * NUM_TRAIN_TIMESTEPS
|
|
timestep = torch.full((b, ), ts, device=self.device, dtype=self.dtype)
|
|
with self._ctx(current_timestep=ts):
|
|
velocity = self.module(hidden_states=hs,
|
|
encoder_hidden_states=ehs,
|
|
timestep=timestep,
|
|
encoder_attention_mask=None)
|
|
# CRITICAL (spec blocker #1): negate so the loop's FLOW_MATCH_STEP reproduces the fastvideo stage's
|
|
# negate-then-scheduler-step. The DiT already casts to float32; _n marshals fp32 -> numpy.
|
|
return self._n(-velocity)
|