Ports the FP4 quantization config + ops from FastVideo-internal
(fastvideo.layers.quantization.fp4_config). flashinfer is imported
lazily inside _require_flashinfer() rather than at module top so
``import fastvideo`` stays cheap on hosts that don't ship flashinfer
— the FP4 kernels themselves still raise a clear ImportError pointing
at ``pip install flashinfer-python`` when invoked.
Behavior is identical to the internal version:
* FP4Config: LTX-2 layer set (48 transformer blocks across attn1/attn2
/audio_to_video_attn/video_to_audio_attn/ffn plus adaln_single.linear),
layer_profile selects between "base" and "refine" stage subsets.
* FP4QuantizeMethod: per-layer quantize_input + apply with stage-aware
routing for refine-only layers (audio_to_video_attn.to_q,
video_to_audio_attn.to_k, video_to_audio_attn.to_v fall through to
dense F.linear during the base stage).
* convert_model_to_fp4(): pre-quantize weight buffers + alpha onto
module attrs at load time so the forward path doesn't pay the
per-step weight-side quantize cost.
Required for Dreamverse integration: dreamverse-server's GPU worker
runs ``pipeline_config.dit_config.quant_config = FP4Config()`` during
LTX-2 model load, which is the dominant inference path.
test_fp4_config covers the lazy-flashinfer contract: import succeeds
without flashinfer; layer_profile round-trips via from_config; calling
into _require_flashinfer raises ImportError with an actionable
message when flashinfer is missing.
Co-authored-by: Junda (David) Su <90978028+Davids048@users.noreply.github.com>
Co-authored-by: Matthew Noto <99706358+RandNMR73@users.noreply.github.com>
Co-authored-by: XOR-op <17672363+XOR-op@users.noreply.github.com>
Co-authored-by: Zhang Peiyuan <42993249+jzhang38@users.noreply.github.com>
