diff --git a/.agents/skills/env-var-conventions/SKILL.md b/.agents/skills/env-var-conventions/SKILL.md index d793281a5..3d12b8f4f 100644 --- a/.agents/skills/env-var-conventions/SKILL.md +++ b/.agents/skills/env-var-conventions/SKILL.md @@ -34,10 +34,15 @@ on a summary here. 1. **Declare or edit the variable in `fastvideo/envs.py`.** - Pick the field type and category that the policy doc lists. - Write a description that states what the variable does and its units. + - To rename, keep the old name in `deprecated_names`. To remove, add the + name to `DEPRECATED_VARIABLES`. Update the uses in `examples/`, + `scripts/`, `docs/`, `apps/`, and the tests. 2. **Read the variable with `envs.NAME.get()` inside a function.** - In tests, change the value with `envs.NAME.override(value)`. - Do not call `os.environ`, `os.getenv`, or `monkeypatch.setenv` for a FastVideo variable. + - To set a variable that another tool reads, call `envs.set_external`, + `envs.setdefault_external`, or `envs.unset_external`. 3. **Regenerate the table in the policy doc.** - Run `python fastvideo/tests/contract/test_env_policy.py`. 4. **Run the contract test.** diff --git a/docs/contributing/activation_trace.md b/docs/contributing/activation_trace.md index 36fc88adc..c3397f901 100644 --- a/docs/contributing/activation_trace.md +++ b/docs/contributing/activation_trace.md @@ -30,7 +30,7 @@ FastVideo and the reference model first produce different numbers?" | General logging | `init_logger(__name__)` | | Per-stage timing | `FASTVIDEO_STAGE_LOGGING` | | Profiling kernel timings | `FASTVIDEO_TORCH_PROFILER_DIR` (see [Profiling](profiling.md)) | -| Function-call tracing | `FASTVIDEO_TRACE_FUNCTION` (heavy) | +| Function-call tracing | `fastvideo.logger.enable_trace_function_call()` (heavy) | ## Quickstart diff --git a/docs/contributing/env_vars.md b/docs/contributing/env_vars.md index a4dac3d29..8f91f1eca 100644 --- a/docs/contributing/env_vars.md +++ b/docs/contributing/env_vars.md @@ -11,14 +11,16 @@ test in the same pull request. 1. **Register every FastVideo variable in `fastvideo/envs.py`.** Each entry declares a type, a default, a category, and a description. Variables that other tools own (CUDA, NCCL, PyTorch, launchers) are not registered; code reads them directly with `os.environ.get("NAME")`, and the name must be in the external-variable allowlist - (`EXTERNAL_ALLOWLIST` in the contract test). + (`EXTERNAL_ALLOWLIST` in the contract test). When FastVideo sets such a variable for the other tool, it calls + `envs.set_external`, `envs.setdefault_external`, or `envs.unset_external`, and the name must be in + `EXTERNAL_WRITE_ALLOWLIST`. 2. **Read with `envs.NAME.get()`, write with `envs.NAME.set()`, and change a value in tests with `envs.NAME.override()`.** Each type has one parsing rule. A value that the rule rejects raises `fastvideo.envs.EnvVarError` instead of falling back to the default. 3. **Name FastVideo variables with the `FASTVIDEO_` prefix.** The second word states the purpose where one applies: `ENABLE_`, `DISABLE_`, `USE_`, `FORCE_`, `DEBUG_`, `TEST_`. 4. **Keep a renamed variable as a deprecated alias until the next minor release.** Setting the old name logs a - warning. Delete a variable that no code reads, and list it in a deprecation table so that setting it logs a + warning. Delete a variable that no code reads, and list it in `DEPRECATED_VARIABLES` so that setting it logs a warning. 5. **Give each setting one source: an argument or an environment variable.** Settings that users change per deployment are arguments (CLI or YAML). Expert switches, emergency off switches, and debugging and test switches @@ -85,6 +87,25 @@ with envs.FASTVIDEO_DEBUG_MY_STAGE.override(True): run_stage() ``` +## Rename or remove a variable + +To rename a variable, declare it under the new name and list the old name in `deprecated_names`: + +```python +FASTVIDEO_LTX2_USE_DISTILLED_SIGMAS = EnvBool(True, + category="sampling", + doc="...", + deprecated_names=("LTX2_USE_DISTILLED_SIGMAS", )) +``` + +`get()` reads an old name only when the new name is unset, and logs a warning once. Update the uses of the old name +in `examples/`, `scripts/`, `docs/`, `apps/`, and the tests in the same pull request. Delete the old name in the next +minor release. + +To remove a variable that no code reads, delete its entry and add the name to `DEPRECATED_VARIABLES` in +`fastvideo/envs.py` with a reason. `FastVideoArgs` calls `envs.warn_deprecated_variables()`, which logs a warning for +each listed variable that is set. Delete the entry in the next minor release. + ## What the contract test checks The test parses every Python file under `fastvideo/`, including `fastvideo/tests/`, with Python's `ast` module. It @@ -100,20 +121,24 @@ It reports each violation as `: `: | | with a name built at runtime (``) | another tool owns, add it to | | | | `EXTERNAL_ALLOWLIST` with a reason. | | `write` | `os.environ[...] = ...`, `setdefault`, `pop`, `del`, | Pass an argument instead. In tests, use | -| | `os.putenv`, `os.unsetenv`, `monkeypatch.setenv`/`delenv` | `envs.NAME.override()`. | +| | `os.putenv`, `os.unsetenv`, `monkeypatch.setenv`/`delenv`, | `envs.NAME.override()`. For a variable that | +| | or an `envs.*_external` call with a name outside | another tool reads, call an | +| | `EXTERNAL_WRITE_ALLOWLIST` | `envs.*_external` helper and add the name to | +| | | `EXTERNAL_WRITE_ALLOWLIST` with a reason. | | `whole-environ` | `os.environ.copy()`, `dict(os.environ)`, iteration, | Read the specific variables that the code | | | `mock.patch.dict(os.environ, ...)`, `os.environ.update` | needs. | | `bare-field` | A registry field used without calling one of its methods, | Call `envs.NAME.get()`. | | | as in `envs.NAME == "auto"` or `getter = envs.NAME.get` | | | `import-time-read` | `envs.NAME.get()` outside a function | Move the read into the function that uses | | | | the value. | -| `prefix` | A registry entry without the `FASTVIDEO_` prefix | Rename the variable and keep the old name as | -| | | a deprecated alias. | -| `unread` | A registry entry that no code reads with `get()` or | Delete the variable and add it to the | -| | `is_set()` | deprecation table. | +| `prefix` | A registry entry without the `FASTVIDEO_` prefix | Rename the variable and keep the old name in | +| | | `deprecated_names`. | +| `unread` | A registry entry that no code reads with `get()` or | Delete the variable and add it to | +| | `is_set()` | `DEPRECATED_VARIABLES`. | -The test recognizes `os` imported under another name and `from os import environ, getenv`. Code that reaches the -environment through `importlib` or `getattr(os, "environ")` is left to code review. +The test recognizes `os` imported under another name, `from os import environ, getenv`, and a name held in a +module-level string constant. Code that reaches the environment through `importlib` or `getattr(os, "environ")` is +left to code review. The test also checks that every registry entry has a category from `envs.CATEGORIES` and a description, and that the table at the end of this page matches the registry. @@ -125,56 +150,93 @@ longer exists also fails the test, so the fixing pull request deletes its entry. ## Registered variables -| Variable | Type | Default | Category | Description | -| ---------------------------------------------- | ------------------------------ | --------------------------- | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `FASTVIDEO_TARGET_DEVICE` | str | `cuda` | build | Target device of FastVideo: cuda, rocm, neuron, cpu, or openvino. | -| `MAX_JOBS` | str | unset | build | Maximum number of parallel compilation jobs. Defaults to the number of CPUs. | -| `NVCC_THREADS` | str | unset | build | Number of nvcc threads. When set, MAX_JOBS is reduced to avoid oversubscribing the CPU. | -| `FASTVIDEO_USE_PRECOMPILED` | bool | `0` | build | Use precompiled binaries (\*.so). | -| `CMAKE_BUILD_TYPE` | str | unset | build | CMake build type: Debug, Release, or RelWithDebInfo. | -| `VERBOSE` | bool | `0` | build | Print verbose logs during installation. | -| `FASTVIDEO_CONFIG_ROOT` | path | computed | path | Root directory for FastVideo configuration files, at runtime and at installation. Defaults to ~/.config/fastvideo, or $XDG_CONFIG_HOME/fastvideo when XDG_CONFIG_HOME is set. | -| `FASTVIDEO_CACHE_ROOT` | path | computed | path | Root directory for FastVideo cache files. Defaults to ~/.cache/fastvideo, or $XDG_CACHE_HOME/fastvideo when XDG_CACHE_HOME is set. | -| `FASTVIDEO_HOST_IP` | str | `""` | distributed | IP address of this node when the node has several network interfaces. Set it on each node for multi-node inference. | -| `FASTVIDEO_LOOPBACK_IP` | str | `""` | distributed | Loopback IP address to use instead of the detected one. | -| `FASTVIDEO_RAY_PER_WORKER_GPUS` | float | `1.0` | distributed | GPUs per Ray worker. A fraction lets Ray schedule several actors on one GPU, so other actors can share the GPUs with FastVideo. | -| `FASTVIDEO_RINGBUFFER_WARNING_INTERVAL` | int | `60` | distributed | Seconds between warnings while the ring buffer is full. | -| `FASTVIDEO_NCCL_SO_PATH` | str | unset | distributed | Path to the NCCL library file. Needed because the nccl>=2.19 that PyTorch ships has a bug (https://github.com/NVIDIA/nccl/issues/1234). | -| `HCCL_SO_PATH` | str | unset | distributed | Path to the HCCL library file on Ascend NPUs. | -| `FASTVIDEO_ENGINE_ITERATION_TIMEOUT_S` | int | `60` | distributed | Timeout in seconds for each engine iteration. | -| `FASTVIDEO_WORKER_MULTIPROC_METHOD` | one of spawn, fork, forkserver | `spawn` | distributed | Multiprocessing start method for worker processes. | -| `FASTVIDEO_ULYSSES_A2A` | one of off, auto | `off` | distributed | Sequence-parallel all-to-all backend. off uses the NCCL path in DistributedAutograd.AllToAll4D. auto uses the fused NVLink kernel when the group is a load-store accessible mesh of 2, 4, 6, or 8 ranks in eager execution, and the NCCL path otherwise. | -| `LD_LIBRARY_PATH` | str | unset | external | Searched for the NCCL library when FASTVIDEO_NCCL_SO_PATH is unset. | -| `LOCAL_RANK` | int | `0` | external | Local rank of the process in a distributed run; selects the GPU device id. | -| `CUDA_VISIBLE_DEVICES` | str | unset | external | Visible devices in a distributed run. | -| `FASTVIDEO_CONFIGURE_LOGGING` | bool | `1` | logging | Configure logging at import. When true, FastVideo uses its default logging configuration or the file in FASTVIDEO_LOGGING_CONFIG_PATH. | -| `FASTVIDEO_LOGGING_CONFIG_PATH` | str | unset | logging | Path to a JSON logging configuration file. | -| `FASTVIDEO_LOGGING_LEVEL` | str | `INFO` | logging | Default logging level. | -| `FASTVIDEO_LOGGING_PREFIX` | str | `""` | logging | Prefix prepended to every log message. | -| `FASTVIDEO_STAGE_LOGGING` | bool | `0` | logging | Log the time that each pipeline stage takes. | -| `FASTVIDEO_ATTENTION_BACKEND` | str | unset | attention | Attention backend, as an AttentionBackendEnum name such as TORCH_SDPA, FLASH_ATTN, VIDEO_SPARSE_ATTN, SAGE_ATTN, or SAGE_ATTN_THREE. FastVideoArgs uses it when FastVideoArgs.attention_backend is unset. | -| `FASTVIDEO_FA4` | bool | `0` | attention | The FLASH_ATTN backend uses FlashAttention-4 (flash_attn.cute) instead of FA3 or FA2. | -| `FASTVIDEO_MINIMAX_H3_FA4_PACKED_VARLEN` | bool | `0` | attention | MiniMax-H3 dense DiT self-attention uses the FlashAttention-4 packed-varlen entry point. This changes the floating-point reduction order, so it is an inference-only opt-in. | -| `FASTVIDEO_VSA_SM100A` | bool | `0` | attention | VIDEO_SPARSE_ATTN_H3 sends no-grad tile-64 forwards to the data-center Blackwell (sm_100a) kernel. fastvideo-kernel reads the same variable with the same rule. | -| `FASTVIDEO_INFERENCE_TORCH_COMPILE` | bool | `0` | performance | Compile each DiT transformer block with fullgraph torch.compile at inference. Same as FastVideoArgs.inference_torch_compile=True. | -| `FASTVIDEO_VAE_PARALLEL_DECODE` | bool | `0` | performance | MiniMax-H3 VAE decode splits its temporal chunks across the sequence-parallel ranks instead of running serially on the output rank. Same as FastVideoArgs.vae_parallel_decode=True. | -| `FASTVIDEO_VAE_PARALLEL_ENCODE` | bool | `0` | performance | MiniMax-H3 reference-video VAE encode splits its temporal chunks across the sequence-parallel ranks. Same as FastVideoArgs.vae_parallel_encode=True. | -| `FASTVIDEO_VAE_PARALLEL_DECODE_STRATEGY` | str | unset | performance | Collective that moves chunks in parallel VAE decode: gather (used when unset) or all_gather. | -| `FASTVIDEO_MINIMAX_H3_FUSIONS` | str | `""` | performance | MiniMax-H3 inference-only Triton fusions: all, 1, or a comma-separated subset of modulate,qknorm_rope,swiglu. Empty, 0, or none keeps the eager implementation. | -| `FASTVIDEO_TEST_DYNAMO_FULLGRAPH_CAPTURE` | bool | `1` | debug | Enable Dynamo fullgraph capture. | -| `FASTVIDEO_NVTX_PROFILE` | bool | `0` | profiling | Emit NVTX ranges for external profilers such as Nsight Systems. | -| `FASTVIDEO_TORCH_PROFILER_DIR` | path | unset | profiling | Enables the torch profiler and sets the directory for its traces. Must be an absolute path. | -| `FASTVIDEO_TORCH_PROFILER_RECORD_SHAPES` | bool | `0` | profiling | Torch profiler records shapes. | -| `FASTVIDEO_TORCH_PROFILER_WITH_PROFILE_MEMORY` | bool | `0` | profiling | Torch profiler profiles memory. | -| `FASTVIDEO_TORCH_PROFILER_WITH_STACK` | bool | `0` | profiling | Torch profiler captures stacks. Costs about 1.5x runtime and 1.4x trace size. | -| `FASTVIDEO_TORCH_PROFILER_WITH_FLOPS` | bool | `0` | profiling | Torch profiler profiles FLOPs. | -| `FASTVIDEO_TORCH_PROFILE_REGIONS` | str | `""` | profiling | Comma-separated profiler regions to record. The torch profiler requires at least one region. | -| `FASTVIDEO_SERVER_DEV_MODE` | bool | `0` | debug | Run the server in development mode with extra debugging endpoints. | -| `FASTVIDEO_TRACE_FUNCTION` | bool | `0` | debug | Trace function calls. | -| `FASTVIDEO_TRACE_ACTIVATIONS` | bool | `0` | debug | Enable activation trace hooks. | -| `FASTVIDEO_TRACE_LAYERS` | str | `""` | debug | Regex filter for traced module names. Empty means all. | -| `FASTVIDEO_TRACE_STATS` | str | `abs_mean,sum` | debug | Comma-separated activation statistics dumped for each output tensor. | -| `FASTVIDEO_TRACE_OUTPUT` | str | `/tmp/fv_trace_.jsonl` | debug | JSONL path for activation traces. The literal <pid> is replaced at runtime. | -| `FASTVIDEO_TRACE_STEPS` | str | `""` | debug | Comma-separated denoising step indices. Empty means all. | -| `FASTVIDEO_CFG_GATE_STEP` | float | `1.0` | sampling | CFG gating fraction in [0, 1]. Steps before len(timesteps) \* X run the conditional and unconditional forwards; later steps reuse the cached difference. 1.0 disables gating. | +| Variable | Type | Default | Category | Description | +| ---------------------------------------------------------- | ------------------------------ | ----------------------------------------------------- | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `FASTVIDEO_CONFIG_ROOT` | path | computed | path | Root directory for FastVideo configuration files, at runtime and at installation. Defaults to ~/.config/fastvideo, or $XDG_CONFIG_HOME/fastvideo when XDG_CONFIG_HOME is set. | +| `FASTVIDEO_CACHE_ROOT` | path | computed | path | Root directory for FastVideo cache files. Defaults to ~/.cache/fastvideo, or $XDG_CACHE_HOME/fastvideo when XDG_CACHE_HOME is set. | +| `FASTVIDEO_REASON1_WEIGHTS_PATH` | str | unset | path | Local path or Hugging Face id of Reason1 weights to load instead of the checkpoint's own. | +| `FASTVIDEO_HOST_IP` | str | `""` | distributed | IP address of this node when the node has several network interfaces. Set it on each node for multi-node inference. | +| `FASTVIDEO_LOOPBACK_IP` | str | `""` | distributed | Loopback IP address to use instead of the detected one. | +| `FASTVIDEO_RAY_PER_WORKER_GPUS` | float | `1.0` | distributed | GPUs per Ray worker. A fraction lets Ray schedule several actors on one GPU, so other actors can share the GPUs with FastVideo. | +| `FASTVIDEO_NCCL_SO_PATH` | str | unset | distributed | Path to the NCCL library file. Needed because the nccl>=2.19 that PyTorch ships has a bug (https://github.com/NVIDIA/nccl/issues/1234). | +| `FASTVIDEO_HCCL_SO_PATH` | str | unset | distributed | Path to the HCCL library file on Ascend NPUs. Deprecated names: `HCCL_SO_PATH`. | +| `FASTVIDEO_WORKER_MULTIPROC_METHOD` | one of spawn, fork, forkserver | `spawn` | distributed | Multiprocessing start method for worker processes. | +| `FASTVIDEO_ULYSSES_A2A` | one of off, auto | `off` | distributed | Sequence-parallel all-to-all backend. off uses the NCCL path in DistributedAutograd.AllToAll4D. auto uses the fused NVLink kernel when the group is a load-store accessible mesh of 2, 4, 6, or 8 ranks in eager execution, and the NCCL path otherwise. | +| `FASTVIDEO_CONFIGURE_LOGGING` | bool | `1` | logging | Configure logging at import. When true, FastVideo uses its default logging configuration or the file in FASTVIDEO_LOGGING_CONFIG_PATH. | +| `FASTVIDEO_LOGGING_CONFIG_PATH` | str | unset | logging | Path to a JSON logging configuration file. | +| `FASTVIDEO_LOGGING_LEVEL` | str | `INFO` | logging | Default logging level. | +| `FASTVIDEO_LOGGING_PREFIX` | str | `""` | logging | Prefix prepended to every log message. | +| `FASTVIDEO_STAGE_LOGGING` | bool | `0` | logging | Log the time that each pipeline stage takes. | +| `FASTVIDEO_ATTENTION_BACKEND` | str | unset | attention | Attention backend, as an AttentionBackendEnum name such as TORCH_SDPA, FLASH_ATTN, VIDEO_SPARSE_ATTN, SAGE_ATTN, or SAGE_ATTN_THREE. FastVideoArgs uses it when FastVideoArgs.attention_backend is unset. | +| `FASTVIDEO_FA4` | bool | `0` | attention | The FLASH_ATTN backend uses FlashAttention-4 (flash_attn.cute) instead of FA3 or FA2. | +| `FASTVIDEO_MINIMAX_H3_FA4_PACKED_VARLEN` | bool | `0` | attention | MiniMax-H3 dense DiT self-attention uses the FlashAttention-4 packed-varlen entry point. This changes the floating-point reduction order, so it is an inference-only opt-in. | +| `FASTVIDEO_VSA_SM100A` | bool | `0` | attention | VIDEO_SPARSE_ATTN_H3 sends no-grad tile-64 forwards to the data-center Blackwell (sm_100a) kernel. fastvideo-kernel reads the same variable with the same rule. | +| `FASTVIDEO_NVFP4_FA4` | bool | `0` | attention | FlashAttention-4 quantizes Q and K to NVFP4. An explicit nvfp4_fa4 attention implementation argument takes precedence. | +| `FASTVIDEO_DISABLE_ATTENTION_COMPILE` | bool | `1` | attention | Keep attention forward out of torch.compile graphs (torch.compiler.disable). Set it to 0 to let attention constructed under that setting be traced. Setting it explicitly to true also blocks regional compile. | +| `FASTVIDEO_MLX_WINDOW` | int | `0` | attention | MLX FastWan windowed attention size in tokens. 0 uses full attention. | +| `FASTVIDEO_MLX_WINDOW_SINK` | int | `0` | attention | Number of sink tokens that MLX windowed attention always attends to. | +| `FASTVIDEO_INFERENCE_TORCH_COMPILE` | bool | `0` | performance | Compile each DiT transformer block with fullgraph torch.compile at inference. Same as FastVideoArgs.inference_torch_compile=True. | +| `FASTVIDEO_VAE_PARALLEL_DECODE` | bool | `0` | performance | MiniMax-H3 VAE decode splits its temporal chunks across the sequence-parallel ranks instead of running serially on the output rank. Same as FastVideoArgs.vae_parallel_decode=True. | +| `FASTVIDEO_VAE_PARALLEL_ENCODE` | bool | `0` | performance | MiniMax-H3 reference-video VAE encode splits its temporal chunks across the sequence-parallel ranks. Same as FastVideoArgs.vae_parallel_encode=True. | +| `FASTVIDEO_VAE_PARALLEL_DECODE_STRATEGY` | str | unset | performance | Collective that moves chunks in parallel VAE decode: gather (used when unset) or all_gather. | +| `FASTVIDEO_MINIMAX_H3_FUSIONS` | str | `""` | performance | MiniMax-H3 inference-only Triton fusions: all, 1, or a comma-separated subset of modulate,qknorm_rope,swiglu. Empty, 0, or none keeps the eager implementation. | +| `FASTVIDEO_FSDP2_AUTOWRAP` | bool | `0` | performance | FSDP2 shards modules by parameter count instead of the model's shard conditions. Not supported by self-forcing distillation. | +| `FASTVIDEO_FSDP2_MIN_PARAMS` | int | `10000000` | performance | Minimum parameter count of a module that FASTVIDEO_FSDP2_AUTOWRAP shards. | +| `FASTVIDEO_MLX_COMPILE` | bool | `0` | performance | Compile the MLX DiT forward with mx.compile. | +| `FASTVIDEO_MLX_FAST_NORM` | bool | `0` | performance | Use MLX fast normalization kernels. | +| `FASTVIDEO_MLX_DQ_GEMM` | str | `1` | performance | MLX dequantized GEMM for affine-quantized weights: 0 turns it off, 1 uses the measured minimum row count, and an integer sets the minimum row count. | +| `FASTVIDEO_LTX2_VAE_CHANNELS_LAST_3D` | bool | `1` | performance | LTX-2 VAE uses the channels_last_3d memory format. | +| `FASTVIDEO_LTX2_DISABLE_AUDIO_AUTOCAST` | bool | `1` | performance | LTX-2 audio decoding runs without CUDA autocast. Deprecated names: `LTX2_DISABLE_AUDIO_AUTOCAST`. | +| `FASTVIDEO_FLUX2_DISABLE_BF16_REDUCED_PRECISION_REDUCTION` | bool | `0` | performance | Flux denoising disables reduced-precision reductions in bf16 matmuls, which tightens accumulation for the 4-step Klein model. | +| `FASTVIDEO_FFMPEG_BIN` | str | `ffmpeg` | output | ffmpeg executable used to save video with audio. | +| `FASTVIDEO_VIDEO_CODEC` | str | `libx264` | output | ffmpeg video codec for saved videos. | +| `FASTVIDEO_NVENC_PRESET` | str | `p1` | output | NVENC preset when the codec is an \*_nvenc codec. | +| `FASTVIDEO_NVENC_TUNE` | str | `ull` | output | NVENC tune option. | +| `FASTVIDEO_NVENC_RC` | str | `constqp` | output | NVENC rate-control mode. | +| `FASTVIDEO_NVENC_QP` | str | `28` | output | NVENC quantization parameter. | +| `FASTVIDEO_NVENC_BF` | str | `0` | output | NVENC number of B-frames. | +| `FASTVIDEO_X264_PRESET` | str | `ultrafast` | output | x264 preset for non-NVENC codecs. | +| `FASTVIDEO_OUTPUT_PIX_FMT` | str | `yuv420p` | output | ffmpeg pixel format for saved videos. | +| `FASTVIDEO_NVTX_PROFILE` | bool | `0` | profiling | Emit NVTX ranges for external profilers such as Nsight Systems. | +| `FASTVIDEO_TORCH_PROFILER_DIR` | path | unset | profiling | Enables the torch profiler and sets the directory for its traces. Must be an absolute path. | +| `FASTVIDEO_TORCH_PROFILER_RECORD_SHAPES` | bool | `0` | profiling | Torch profiler records shapes. | +| `FASTVIDEO_TORCH_PROFILER_WITH_PROFILE_MEMORY` | bool | `0` | profiling | Torch profiler profiles memory. | +| `FASTVIDEO_TORCH_PROFILER_WITH_STACK` | bool | `0` | profiling | Torch profiler captures stacks. Costs about 1.5x runtime and 1.4x trace size. | +| `FASTVIDEO_TORCH_PROFILER_WITH_FLOPS` | bool | `0` | profiling | Torch profiler profiles FLOPs. | +| `FASTVIDEO_TORCH_PROFILE_REGIONS` | str | `""` | profiling | Comma-separated profiler regions to record. The torch profiler requires at least one region. | +| `FASTVIDEO_TRACE_ACTIVATIONS` | bool | `0` | debug | Enable activation trace hooks. | +| `FASTVIDEO_TRACE_LAYERS` | str | `""` | debug | Regex filter for traced module names. Empty means all. | +| `FASTVIDEO_TRACE_STATS` | str | `abs_mean,sum` | debug | Comma-separated activation statistics dumped for each output tensor. | +| `FASTVIDEO_TRACE_OUTPUT` | str | `/tmp/fv_trace_.jsonl` | debug | JSONL path for activation traces. The literal <pid> is replaced at runtime. | +| `FASTVIDEO_TRACE_STEPS` | str | `""` | debug | Comma-separated denoising step indices. Empty means all. | +| `FASTVIDEO_H3_VSA_PROBE` | str | unset | debug | Output directory for the VSA-H3 attention-mass probe, which writes one .pt file per step, layer, and rank. Keeps the model out of regional compile. | +| `FASTVIDEO_LTX2_GEMMA_LOG` | str | `""` | debug | Log file for LTX-2 Gemma text-encoder hidden states, used by parity tests. Deprecated names: `LTX2_FASTVIDEO_GEMMA_LOG`. | +| `FASTVIDEO_COSMOS25_LOG_KNOBS` | bool | `0` | debug | Log the Cosmos 2.5 latent-preparation conditioning inputs. | +| `FASTVIDEO_CFG_GATE_STEP` | float | `1.0` | sampling | CFG gating fraction in [0, 1]. Steps before len(timesteps) \* X run the conditional and unconditional forwards; later steps reuse the cached difference. 1.0 disables gating. | +| `FASTVIDEO_LTX2_USE_DISTILLED_SIGMAS` | bool | `1` | sampling | LTX-2 uses the distilled sigma schedule when FastVideoArgs.ltx2_use_distilled_sigmas is also true. Deprecated names: `LTX2_USE_DISTILLED_SIGMAS`. | +| `FASTVIDEO_EVAL_CACHE` | path | computed | eval | Cache directory for evaluation models and datasets. Defaults to $FASTVIDEO_CACHE_ROOT/eval. | +| `FASTVIDEO_PHYSICS_IQ_BUCKET_URL` | str | `https://storage.googleapis.com/physics-iq-benchmark` | eval | Base URL of the Physics-IQ benchmark bucket. | +| `FASTVIDEO_VBENCH_FULL_INFO_JSON` | str | unset | eval | Path to VBench_full_info.json, used instead of the vendored copy. Deprecated names: `VBENCH_FULL_INFO_JSON`. | +| `FASTVIDEO_FVD_REF_FEATURES` | str | unset | eval | Cached reference-feature file for the FVD metric. | +| `FASTVIDEO_FAD_REF_FEATURES` | str | unset | eval | Cached reference-feature file for the audio Frechet distance metric. | +| `FASTVIDEO_TEST_LTX2_OVERFIT_DATA_DIR` | str | `data/cats` | test | Raw data directory for preprocess_ltx2_overfit.py. Deprecated names: `LTX2_OVERFIT_DATA_DIR`. | +| `FASTVIDEO_TEST_LTX2_OVERFIT_CAPTION_JSON` | str | `videos2caption_1_sample.json` | test | Caption file, relative to the raw data directory. Deprecated names: `LTX2_OVERFIT_CAPTION_JSON`. | +| `FASTVIDEO_TEST_LTX2_OVERFIT_VIDEO_SUBDIR` | str | `video` | test | Video subdirectory, relative to the raw data directory. Deprecated names: `LTX2_OVERFIT_VIDEO_SUBDIR`. | +| `FASTVIDEO_TEST_LTX2_OVERFIT_OUTPUT_DIR` | str | `data/ltx2_overfit_preprocessed` | test | Output directory for preprocess_ltx2_overfit.py. Deprecated names: `LTX2_OVERFIT_OUTPUT_DIR`. | +| `FASTVIDEO_TEST_LTX2_OVERFIT_MODEL` | str | `FastVideo/LTX2-Distilled-Diffusers` | test | Model repository whose encoders preprocess_ltx2_overfit.py uses. Deprecated names: `LTX2_OVERFIT_MODEL`. | +| `FASTVIDEO_TEST_LTX2_OVERFIT_NUM_COPIES` | int | `4` | test | Number of copies of the overfit sample in the parquet file. Deprecated names: `LTX2_OVERFIT_NUM_COPIES`. | +| `FASTVIDEO_TEST_KANDINSKY5_OVERFIT_DATA_DIR` | str | `data/kandinsky5_overfit` | test | Raw data directory for preprocess_kandinsky5_overfit.py. Deprecated names: `KANDINSKY5_OVERFIT_DATA_DIR`. | +| `FASTVIDEO_TEST_KANDINSKY5_OVERFIT_OUTPUT_DIR` | str | `data/kandinsky5_overfit_preprocessed` | test | Output directory for preprocess_kandinsky5_overfit.py. Deprecated names: `KANDINSKY5_OVERFIT_OUTPUT_DIR`. | + +Variables that FastVideo no longer reads; setting one logs a warning: + +| Deprecated variable | Reason | +| ----------------------------------------- | ---------------- | +| `FASTVIDEO_TARGET_DEVICE` | no code reads it | +| `FASTVIDEO_USE_PRECOMPILED` | no code reads it | +| `FASTVIDEO_RINGBUFFER_WARNING_INTERVAL` | no code reads it | +| `FASTVIDEO_ENGINE_ITERATION_TIMEOUT_S` | no code reads it | +| `FASTVIDEO_SERVER_DEV_MODE` | no code reads it | +| `FASTVIDEO_TEST_DYNAMO_FULLGRAPH_CAPTURE` | no code reads it | +| `FASTVIDEO_TRACE_FUNCTION` | no code reads it | diff --git a/examples/train/configs/fine_tuning/kandinsky5/README.md b/examples/train/configs/fine_tuning/kandinsky5/README.md index 5205128c0..5f8a80043 100644 --- a/examples/train/configs/fine_tuning/kandinsky5/README.md +++ b/examples/train/configs/fine_tuning/kandinsky5/README.md @@ -27,7 +27,7 @@ layout) with: ```bash # from the repo root; reads data/kandinsky5_overfit and writes # data/kandinsky5_overfit_preprocessed by default -- override with -# KANDINSKY5_OVERFIT_DATA_DIR / KANDINSKY5_OVERFIT_OUTPUT_DIR env vars +# FASTVIDEO_TEST_KANDINSKY5_OVERFIT_DATA_DIR / FASTVIDEO_TEST_KANDINSKY5_OVERFIT_OUTPUT_DIR env vars python -m fastvideo.pipelines.preprocess.preprocess_kandinsky5_overfit ``` diff --git a/examples/train/configs/fine_tuning/ltx2_3/nvfp4_qat_t2v.yaml b/examples/train/configs/fine_tuning/ltx2_3/nvfp4_qat_t2v.yaml index 00993ef70..26a72b8af 100644 --- a/examples/train/configs/fine_tuning/ltx2_3/nvfp4_qat_t2v.yaml +++ b/examples/train/configs/fine_tuning/ltx2_3/nvfp4_qat_t2v.yaml @@ -32,7 +32,7 @@ # (see docs/training/data_preprocess.md; # fastvideo/pipelines/preprocess/preprocess_ltx2_overfit.py is a minimal # LTX-2 reference to adapt — set -# LTX2_OVERFIT_MODEL=FastVideo/LTX-2.3-Distilled-Diffusers). Data paths, +# FASTVIDEO_TEST_LTX2_OVERFIT_MODEL=FastVideo/LTX-2.3-Distilled-Diffusers). Data paths, # step counts, and learning rate below are placeholders you must adapt # to your workload. # diff --git a/examples/train/configs/fine_tuning/ltx2_3/t2v.yaml b/examples/train/configs/fine_tuning/ltx2_3/t2v.yaml index 71a5fbc12..0e69887de 100644 --- a/examples/train/configs/fine_tuning/ltx2_3/t2v.yaml +++ b/examples/train/configs/fine_tuning/ltx2_3/t2v.yaml @@ -12,7 +12,7 @@ # (see docs/training/data_preprocess.md; # fastvideo/pipelines/preprocess/preprocess_ltx2_overfit.py is a minimal # LTX-2 reference to adapt — set -# LTX2_OVERFIT_MODEL=FastVideo/LTX-2.3-Distilled-Diffusers). +# FASTVIDEO_TEST_LTX2_OVERFIT_MODEL=FastVideo/LTX-2.3-Distilled-Diffusers). # # Run: # NUM_GPUS=4 bash examples/train/run.sh \ diff --git a/examples/train/configs/overfit_ltx2_3_t2v.yaml b/examples/train/configs/overfit_ltx2_3_t2v.yaml index b86e29533..5ffd47c17 100644 --- a/examples/train/configs/overfit_ltx2_3_t2v.yaml +++ b/examples/train/configs/overfit_ltx2_3_t2v.yaml @@ -8,8 +8,8 @@ # # Preprocess data first (writes data/ltx2_3_overfit_preprocessed): # CUDA_VISIBLE_DEVICES=0 \ -# LTX2_OVERFIT_MODEL=FastVideo/LTX-2.3-Distilled-Diffusers \ -# LTX2_OVERFIT_OUTPUT_DIR=data/ltx2_3_overfit_preprocessed \ +# FASTVIDEO_TEST_LTX2_OVERFIT_MODEL=FastVideo/LTX-2.3-Distilled-Diffusers \ +# FASTVIDEO_TEST_LTX2_OVERFIT_OUTPUT_DIR=data/ltx2_3_overfit_preprocessed \ # python fastvideo/pipelines/preprocess/preprocess_ltx2_overfit.py # # Run: diff --git a/fastvideo/attention/backends/flash_attn.py b/fastvideo/attention/backends/flash_attn.py index 99e6f7c70..5ff725a40 100644 --- a/fastvideo/attention/backends/flash_attn.py +++ b/fastvideo/attention/backends/flash_attn.py @@ -5,6 +5,7 @@ import torch import torch.nn.functional as F from dataclasses import dataclass +import fastvideo.envs as envs from fastvideo.attention.utils.flash_attn_default import ( fa_version, flash_attn_func_compilable, @@ -264,7 +265,7 @@ class FlashAttentionImpl(AttentionImpl): # keeps its previous semantics. nvfp4_fa4 = extra_impl_args.get("nvfp4_fa4") if nvfp4_fa4 is None: - nvfp4_fa4 = os.environ.get("FASTVIDEO_NVFP4_FA4", "0") == "1" + nvfp4_fa4 = envs.FASTVIDEO_NVFP4_FA4.get() self.nvfp4_fa4 = bool(nvfp4_fa4) if self.nvfp4_fa4: cap = torch.cuda.get_device_capability() diff --git a/fastvideo/attention/backends/video_sparse_attn_h3_probe.py b/fastvideo/attention/backends/video_sparse_attn_h3_probe.py index ff146e605..8084bd4fc 100644 --- a/fastvideo/attention/backends/video_sparse_attn_h3_probe.py +++ b/fastvideo/attention/backends/video_sparse_attn_h3_probe.py @@ -23,12 +23,14 @@ import os import torch +import fastvideo.envs as envs + _FRACS = (0.05, 0.10, 0.125, 0.25, 0.50, 0.75) _TRUE_ROWS = 128 # sampled query rows per layer for the token-true check def probe_enabled() -> str | None: - return os.environ.get("FASTVIDEO_H3_VSA_PROBE") or None + return envs.FASTVIDEO_H3_VSA_PROBE.get() or None @torch.no_grad() diff --git a/fastvideo/attention/layer.py b/fastvideo/attention/layer.py index ac53492b7..ff499e452 100644 --- a/fastvideo/attention/layer.py +++ b/fastvideo/attention/layer.py @@ -1,11 +1,11 @@ # SPDX-License-Identifier: Apache-2.0 -import os from functools import wraps import torch import torch.nn as nn +import fastvideo.envs as envs from fastvideo.attention.selector import backend_name_to_enum, get_attn_backend from fastvideo.distributed.communication_op import (sequence_model_parallel_all_gather, sequence_model_parallel_all_to_all_4D) @@ -23,10 +23,7 @@ def _attention_compile_disabled() -> bool: ``torch.compiler.disable``). Set ``FASTVIDEO_DISABLE_ATTENTION_COMPILE=0`` to let attention instances constructed under that environment be traced. """ - val = os.environ.get("FASTVIDEO_DISABLE_ATTENTION_COMPILE") - if val is None: - return True - return val.strip().lower() not in ("0", "false", "no", "off", "") + return envs.FASTVIDEO_DISABLE_ATTENTION_COMPILE.get() def _attention_compile_explicitly_disabled() -> bool: @@ -35,7 +32,7 @@ def _attention_compile_explicitly_disabled() -> bool: Regional compile can override the historical default for one loaded transformer, but it must still honor an explicit debugging escape hatch. """ - return "FASTVIDEO_DISABLE_ATTENTION_COMPILE" in os.environ and _attention_compile_disabled() + return envs.FASTVIDEO_DISABLE_ATTENTION_COMPILE.is_set() and _attention_compile_disabled() def _maybe_compiler_disable(fn): diff --git a/fastvideo/attention/selector.py b/fastvideo/attention/selector.py index 302dd4683..76202d342 100644 --- a/fastvideo/attention/selector.py +++ b/fastvideo/attention/selector.py @@ -1,7 +1,6 @@ # SPDX-License-Identifier: Apache-2.0 # Adapted from vllm: https://github.com/vllm-project/vllm/blob/v0.7.3/vllm/attention/selector.py -import os from collections.abc import Generator from contextlib import contextmanager from contextvars import ContextVar @@ -15,7 +14,7 @@ import fastvideo.envs as envs from fastvideo.attention.backends.abstract import AttentionBackend from fastvideo.logger import init_logger from fastvideo.platforms import AttentionBackendEnum -from fastvideo.utils import STR_BACKEND_ENV_VAR, resolve_obj_by_qualname +from fastvideo.utils import resolve_obj_by_qualname logger = init_logger(__name__) @@ -65,7 +64,7 @@ def get_env_variable_attn_backend() -> AttentionBackendEnum | None: * _Backend enum value if an override is specified * None otherwise ''' - backend_name = os.environ.get(STR_BACKEND_ENV_VAR) + backend_name = envs.FASTVIDEO_ATTENTION_BACKEND.get() return (None if backend_name is None else backend_name_to_enum(backend_name)) diff --git a/fastvideo/benchmarks/mlx_fastwan_bench.py b/fastvideo/benchmarks/mlx_fastwan_bench.py index 15a096d3f..3e0bf8e54 100644 --- a/fastvideo/benchmarks/mlx_fastwan_bench.py +++ b/fastvideo/benchmarks/mlx_fastwan_bench.py @@ -46,6 +46,7 @@ from examples.inference.basic.mlx_wan_prompt_to_video import ( encode_prompt, make_rotary_embeddings, ) +import fastvideo.envs as envs from fastvideo.mlx_runtime.memory import add_memory_limit_args, apply_memory_limits, cleanup_mlx # The highest-fidelity cell; used as the default SSIM reference when no external @@ -524,8 +525,8 @@ def _generate_cell( "peak_gib": max(load_peak, denoise_peak) / (1024**3), "quantization": quantization or "none", "compute_dtype": base_dtype, - "compile": os.environ.get("FASTVIDEO_MLX_COMPILE", "0") == "1", - "fast_norm": os.environ.get("FASTVIDEO_MLX_FAST_NORM", "0") == "1", + "compile": envs.FASTVIDEO_MLX_COMPILE.get(), + "fast_norm": envs.FASTVIDEO_MLX_FAST_NORM.get(), "mlx_memory_limit_gib": args.mlx_memory_limit_gib, "mlx_cache_limit_gib": args.mlx_cache_limit_gib, "mlx_disable_cache": args.mlx_disable_cache, diff --git a/fastvideo/distributed/device_communicators/pyhccl_wrapper.py b/fastvideo/distributed/device_communicators/pyhccl_wrapper.py index dfc98a5dd..a582fb83b 100644 --- a/fastvideo/distributed/device_communicators/pyhccl_wrapper.py +++ b/fastvideo/distributed/device_communicators/pyhccl_wrapper.py @@ -144,7 +144,7 @@ class HCCLLibrary: "Otherwise, the hccl library might not exist, be corrupted " "or it does not support the current platform %s. " "If you already have the library, please set the " - "environment variable HCCL_SO_PATH" + "environment variable FASTVIDEO_HCCL_SO_PATH" " to point to the correct hccl library path.", so_file, platform.platform()) raise e diff --git a/fastvideo/distributed/parallel_state.py b/fastvideo/distributed/parallel_state.py index 49b2e4f34..dcf150dd1 100644 --- a/fastvideo/distributed/parallel_state.py +++ b/fastvideo/distributed/parallel_state.py @@ -40,7 +40,6 @@ import torch.distributed import torch.distributed as dist from torch.distributed import Backend, ProcessGroup, ReduceOp -import fastvideo.envs as envs from fastvideo.distributed.device_communicators.base_device_communicator import (DeviceCommunicatorBase) from fastvideo.distributed.device_communicators.cpu_communicator import (CpuCommunicator) from fastvideo.distributed.utils import StatelessProcessGroup @@ -759,7 +758,7 @@ def init_distributed_environment( if local_rank == -1: # local rank not set, this usually happens in single-node # setting, where we can use rank as local rank - local_rank = envs.LOCAL_RANK.get() if distributed_init_method == "env://" else rank + local_rank = int(os.environ.get("LOCAL_RANK", "0")) if distributed_init_method == "env://" else rank global _WORLD if _WORLD is None: ranks = list(range(torch.distributed.get_world_size())) @@ -884,9 +883,9 @@ def get_local_torch_device() -> torch.device: """Return the torch device for the current rank.""" from fastvideo.platforms import current_platform if current_platform.is_npu(): - device = torch.device(f"npu:{envs.LOCAL_RANK.get()}") + device = torch.device(f"npu:{int(os.environ.get('LOCAL_RANK', '0'))}") elif current_platform.is_cuda_alike() or current_platform.is_cuda(): - device = torch.device(f"cuda:{envs.LOCAL_RANK.get()}") + device = torch.device(f"cuda:{int(os.environ.get('LOCAL_RANK', '0'))}") else: device = torch.device("mps") return device diff --git a/fastvideo/entrypoints/streaming/worker.py b/fastvideo/entrypoints/streaming/worker.py index 9cfe1e6bb..d90ddb2a3 100644 --- a/fastvideo/entrypoints/streaming/worker.py +++ b/fastvideo/entrypoints/streaming/worker.py @@ -50,9 +50,9 @@ def worker_main( ``VideoGenerator`` construction + generation happens here, not in the parent's event loop. """ - import os + import fastvideo.envs as envs - os.environ["CUDA_VISIBLE_DEVICES"] = str(gpu_id) + envs.set_external("CUDA_VISIBLE_DEVICES", str(gpu_id)) try: from fastvideo import VideoGenerator diff --git a/fastvideo/entrypoints/video_generator.py b/fastvideo/entrypoints/video_generator.py index 410f67c54..2f8468a68 100644 --- a/fastvideo/entrypoints/video_generator.py +++ b/fastvideo/entrypoints/video_generator.py @@ -26,6 +26,7 @@ import torch import torchvision from einops import rearrange +import fastvideo.envs as envs from fastvideo.api.compat import ( REQUEST_BATCH_EXTRA_PASSTHROUGH_FIELDS, expand_request_prompt_batch, @@ -200,7 +201,7 @@ class VideoGenerator: if kwargs.pop("nvfp4_fa4", False): import os os.environ["FASTVIDEO_NVFP4_FA4"] = "1" - os.environ.setdefault("CUTE_DSL_ENABLE_TVM_FFI", "1") + envs.setdefault_external("CUTE_DSL_ENABLE_TVM_FFI", "1") typed_config = kwargs.pop("config", None) if typed_config is not None: if model_path is not None: @@ -1191,7 +1192,7 @@ class VideoGenerator: sample_rate: int, ) -> bool: """Encode video+audio using ffmpeg via rawvideo stdin + WAV input.""" - ffmpeg_bin = shutil.which(os.getenv("FASTVIDEO_FFMPEG_BIN", "ffmpeg")) + ffmpeg_bin = shutil.which(envs.FASTVIDEO_FFMPEG_BIN.get()) if ffmpeg_bin is None: logger.warning("ffmpeg not found; cannot use ffmpeg pipe save.") return False @@ -1201,7 +1202,7 @@ class VideoGenerator: height = int(frames[0].shape[0]) width = int(frames[0].shape[1]) - codec = os.getenv("FASTVIDEO_VIDEO_CODEC", "libx264") + codec = envs.FASTVIDEO_VIDEO_CODEC.get() try: with tempfile.TemporaryDirectory() as tmpdir: @@ -1236,23 +1237,23 @@ class VideoGenerator: if codec.endswith("_nvenc"): nvenc_options = [ - ("preset", os.getenv("FASTVIDEO_NVENC_PRESET", "p1")), - ("tune", os.getenv("FASTVIDEO_NVENC_TUNE", "ull")), - ("rc", os.getenv("FASTVIDEO_NVENC_RC", "constqp")), - ("qp", os.getenv("FASTVIDEO_NVENC_QP", "28")), - ("bf", os.getenv("FASTVIDEO_NVENC_BF", "0")), + ("preset", envs.FASTVIDEO_NVENC_PRESET.get()), + ("tune", envs.FASTVIDEO_NVENC_TUNE.get()), + ("rc", envs.FASTVIDEO_NVENC_RC.get()), + ("qp", envs.FASTVIDEO_NVENC_QP.get()), + ("bf", envs.FASTVIDEO_NVENC_BF.get()), ] for option_name, option_value in nvenc_options: if cls._ffmpeg_encoder_supports_option(ffmpeg_bin, codec, option_name): cmd += [f"-{option_name}", option_value] else: - cmd += ["-preset", os.getenv("FASTVIDEO_X264_PRESET", "ultrafast")] + cmd += ["-preset", envs.FASTVIDEO_X264_PRESET.get()] cmd += [ "-c:a", "aac", "-pix_fmt", - os.getenv("FASTVIDEO_OUTPUT_PIX_FMT", "yuv420p"), + envs.FASTVIDEO_OUTPUT_PIX_FMT.get(), # Audio and video decoders may produce different durations. # Preserve every video frame and pad/trim audio to match. "-af", diff --git a/fastvideo/envs.py b/fastvideo/envs.py index bfa8cd809..881b2f158 100644 --- a/fastvideo/envs.py +++ b/fastvideo/envs.py @@ -12,6 +12,7 @@ The policy for environment variables is in ``docs/contributing/env_vars.md``, and ``fastvideo/tests/contract/test_env_policy.py`` enforces it. """ +import logging import os from collections.abc import Callable, Iterator from contextlib import contextmanager @@ -24,12 +25,23 @@ S = TypeVar("S", str, str | None) POLICY_DOC = "docs/contributing/env_vars.md" # Allowed values of EnvField.category. -CATEGORIES = ("build", "path", "distributed", "external", "logging", "attention", "performance", "profiling", "debug", - "sampling") +CATEGORIES = ("path", "distributed", "logging", "attention", "performance", "output", "profiling", "debug", "sampling", + "eval", "test") _TRUE_VALUES = frozenset({"1", "true", "yes", "on"}) _FALSE_VALUES = frozenset({"0", "false", "no", "off", ""}) +# fastvideo.logger imports this module, so warnings go through the standard +# logging module; the "fastvideo" logger configuration still applies. +_logger = logging.getLogger(__name__) +_warned_messages: set[str] = set() + + +def _warn_once(message: str) -> None: + if message not in _warned_messages: + _warned_messages.add(message) + _logger.warning(message) + class EnvVarError(ValueError): """An environment variable holds a value that its registered type rejects.""" @@ -39,16 +51,20 @@ class EnvField(Generic[T]): """One registered environment variable: its type, default, category, and description. ``default`` is either the value itself or a zero-argument function that - computes it on each read while the variable is unset. + computes it on each read while the variable is unset. ``deprecated_names`` + lists earlier names of a renamed variable; they are read, with a warning, + only when the variable itself is unset. """ type_name = "" - def __init__(self, default: T | Callable[[], T], *, category: str, doc: str) -> None: + def __init__(self, default: T | Callable[[], T], *, category: str, doc: str, + deprecated_names: tuple[str, ...] = ()) -> None: self.name = "" # Set by _register_fields() from the module attribute name. self.default = default self.category = category self.doc = doc + self.deprecated_names = deprecated_names def parse(self, raw: str) -> T: raise NotImplementedError @@ -56,40 +72,55 @@ class EnvField(Generic[T]): def format(self, value: T) -> str: return str(value) + def _read_raw(self) -> tuple[str, str] | None: + """Return (name, raw value) from the variable or its first set deprecated name.""" + raw = os.environ.get(self.name) + if raw is not None: + return self.name, raw + for old_name in self.deprecated_names: + raw = os.environ.get(old_name) + if raw is not None: + _warn_once(f"{old_name} is deprecated and will be removed in the next minor release; " + f"set {self.name} instead.") + return old_name, raw + return None + def get(self) -> T: """Return the parsed value, or the default when the variable is unset.""" - raw = os.environ.get(self.name) - if raw is None: + found = self._read_raw() + if found is None: return self.default() if callable(self.default) else self.default + name, raw = found try: return self.parse(raw) except ValueError as exc: - raise EnvVarError(f"Invalid value {raw!r} for {self.name}: {exc}. See {POLICY_DOC}.") from None + raise EnvVarError(f"Invalid value {raw!r} for {name}: {exc}. See {POLICY_DOC}.") from None def is_set(self) -> bool: - return self.name in os.environ + return any(name in os.environ for name in (self.name, *self.deprecated_names)) def set(self, value: T) -> None: os.environ[self.name] = self.format(value) def clear(self) -> None: - os.environ.pop(self.name, None) + for name in (self.name, *self.deprecated_names): + os.environ.pop(name, None) @contextmanager def override(self, value: T | None) -> Iterator[None]: - """Set the variable, or unset it when ``value`` is None, and restore the previous value on exit.""" - previous = os.environ.get(self.name) - if value is None: - self.clear() - else: + """Set the variable, or unset it when ``value`` is None, and restore the previous values on exit.""" + previous = {name: os.environ.get(name) for name in (self.name, *self.deprecated_names)} + self.clear() + if value is not None: self.set(value) try: yield finally: - if previous is None: - self.clear() - else: - os.environ[self.name] = previous + for name, old_value in previous.items(): + if old_value is None: + os.environ.pop(name, None) + else: + os.environ[name] = old_value def __bool__(self) -> bool: raise TypeError(f"Use envs.{self.name}.get() to read {self.name}.") @@ -171,20 +202,25 @@ def get_default_config_root() -> str: ) -# ================== Installation ================== +# Variables that other tools read (CUDA, NCCL, PyTorch, launchers) are written +# only through these helpers. The contract test checks each name against its +# write allowlist. + + +def set_external(name: str, value: str) -> None: + """Set an environment variable that another tool reads.""" + os.environ[name] = value + + +def setdefault_external(name: str, value: str) -> None: + """Set an environment variable that another tool reads, unless it is already set.""" + os.environ.setdefault(name, value) + + +def unset_external(name: str) -> None: + """Remove an environment variable that another tool reads.""" + os.environ.pop(name, None) -FASTVIDEO_TARGET_DEVICE = EnvStr("cuda", - category="build", - doc="Target device of FastVideo: cuda, rocm, neuron, cpu, or openvino.") -MAX_JOBS = EnvStr(None, - category="build", - doc="Maximum number of parallel compilation jobs. Defaults to the number of CPUs.") -NVCC_THREADS = EnvStr(None, - category="build", - doc="Number of nvcc threads. When set, MAX_JOBS is reduced to avoid oversubscribing the CPU.") -FASTVIDEO_USE_PRECOMPILED = EnvBool(False, category="build", doc="Use precompiled binaries (*.so).") -CMAKE_BUILD_TYPE = EnvStr(None, category="build", doc="CMake build type: Debug, Release, or RelWithDebInfo.") -VERBOSE = EnvBool(False, category="build", doc="Print verbose logs during installation.") # ================== Paths ================== @@ -198,6 +234,10 @@ FASTVIDEO_CACHE_ROOT = EnvPath( category="path", doc="Root directory for FastVideo cache files. " "Defaults to ~/.cache/fastvideo, or $XDG_CACHE_HOME/fastvideo when XDG_CACHE_HOME is set.") +FASTVIDEO_REASON1_WEIGHTS_PATH = EnvStr(None, + category="path", + doc="Local path or Hugging Face id of Reason1 weights to load instead of the " + "checkpoint's own.") # ================== Distributed ================== @@ -214,18 +254,15 @@ FASTVIDEO_RAY_PER_WORKER_GPUS = EnvFloat( category="distributed", doc="GPUs per Ray worker. A fraction lets Ray schedule several actors on one GPU, so other actors can share " "the GPUs with FastVideo.") -FASTVIDEO_RINGBUFFER_WARNING_INTERVAL = EnvInt(60, - category="distributed", - doc="Seconds between warnings while the ring buffer is full.") FASTVIDEO_NCCL_SO_PATH = EnvStr( None, category="distributed", doc="Path to the NCCL library file. Needed because the nccl>=2.19 that PyTorch ships has a bug " "(https://github.com/NVIDIA/nccl/issues/1234).") -HCCL_SO_PATH = EnvStr(None, category="distributed", doc="Path to the HCCL library file on Ascend NPUs.") -FASTVIDEO_ENGINE_ITERATION_TIMEOUT_S = EnvInt(60, - category="distributed", - doc="Timeout in seconds for each engine iteration.") +FASTVIDEO_HCCL_SO_PATH = EnvStr(None, + category="distributed", + doc="Path to the HCCL library file on Ascend NPUs.", + deprecated_names=("HCCL_SO_PATH", )) FASTVIDEO_WORKER_MULTIPROC_METHOD = EnvChoice("spawn", choices=("spawn", "fork", "forkserver"), category="distributed", @@ -238,18 +275,6 @@ FASTVIDEO_ULYSSES_A2A = EnvChoice( "the fused NVLink kernel when the group is a load-store accessible mesh of 2, 4, 6, or 8 ranks in eager " "execution, and the NCCL path otherwise.") -# ================== External variables ================== -# Variables that other tools set. They stay registered so that Ray copies them -# to its workers until the external-variable allowlist replaces them. - -LD_LIBRARY_PATH = EnvStr(None, - category="external", - doc="Searched for the NCCL library when FASTVIDEO_NCCL_SO_PATH is unset.") -LOCAL_RANK = EnvInt(0, - category="external", - doc="Local rank of the process in a distributed run; selects the GPU device id.") -CUDA_VISIBLE_DEVICES = EnvStr(None, category="external", doc="Visible devices in a distributed run.") - # ================== Logging ================== FASTVIDEO_CONFIGURE_LOGGING = EnvBool( @@ -285,6 +310,22 @@ FASTVIDEO_VSA_SM100A = EnvBool( category="attention", doc="VIDEO_SPARSE_ATTN_H3 sends no-grad tile-64 forwards to the data-center Blackwell (sm_100a) kernel. " "fastvideo-kernel reads the same variable with the same rule.") +FASTVIDEO_NVFP4_FA4 = EnvBool( + False, + category="attention", + doc="FlashAttention-4 quantizes Q and K to NVFP4. An explicit nvfp4_fa4 attention implementation argument " + "takes precedence.") +FASTVIDEO_DISABLE_ATTENTION_COMPILE = EnvBool( + True, + category="attention", + doc="Keep attention forward out of torch.compile graphs (torch.compiler.disable). Set it to 0 to let attention " + "constructed under that setting be traced. Setting it explicitly to true also blocks regional compile.") +FASTVIDEO_MLX_WINDOW = EnvInt(0, + category="attention", + doc="MLX FastWan windowed attention size in tokens. 0 uses full attention.") +FASTVIDEO_MLX_WINDOW_SINK = EnvInt(0, + category="attention", + doc="Number of sink tokens that MLX windowed attention always attends to.") # ================== Performance ================== @@ -316,7 +357,44 @@ FASTVIDEO_MINIMAX_H3_FUSIONS = EnvStr( category="performance", doc="MiniMax-H3 inference-only Triton fusions: all, 1, or a comma-separated subset of " "modulate,qknorm_rope,swiglu. Empty, 0, or none keeps the eager implementation.") -FASTVIDEO_TEST_DYNAMO_FULLGRAPH_CAPTURE = EnvBool(True, category="debug", doc="Enable Dynamo fullgraph capture.") +FASTVIDEO_FSDP2_AUTOWRAP = EnvBool(False, + category="performance", + doc="FSDP2 shards modules by parameter count instead of the model's shard " + "conditions. Not supported by self-forcing distillation.") +FASTVIDEO_FSDP2_MIN_PARAMS = EnvInt(10000000, + category="performance", + doc="Minimum parameter count of a module that FASTVIDEO_FSDP2_AUTOWRAP shards.") +FASTVIDEO_MLX_COMPILE = EnvBool(False, category="performance", doc="Compile the MLX DiT forward with mx.compile.") +FASTVIDEO_MLX_FAST_NORM = EnvBool(False, category="performance", doc="Use MLX fast normalization kernels.") +FASTVIDEO_MLX_DQ_GEMM = EnvStr( + "1", + category="performance", + doc="MLX dequantized GEMM for affine-quantized weights: 0 turns it off, 1 uses the measured minimum row count, " + "and an integer sets the minimum row count.") +FASTVIDEO_LTX2_VAE_CHANNELS_LAST_3D = EnvBool(True, + category="performance", + doc="LTX-2 VAE uses the channels_last_3d memory format.") +FASTVIDEO_LTX2_DISABLE_AUDIO_AUTOCAST = EnvBool(True, + category="performance", + doc="LTX-2 audio decoding runs without CUDA autocast.", + deprecated_names=("LTX2_DISABLE_AUDIO_AUTOCAST", )) +FASTVIDEO_FLUX2_DISABLE_BF16_REDUCED_PRECISION_REDUCTION = EnvBool( + False, + category="performance", + doc="Flux denoising disables reduced-precision reductions in bf16 matmuls, which tightens accumulation for " + "the 4-step Klein model.") + +# ================== Output encoding ================== + +FASTVIDEO_FFMPEG_BIN = EnvStr("ffmpeg", category="output", doc="ffmpeg executable used to save video with audio.") +FASTVIDEO_VIDEO_CODEC = EnvStr("libx264", category="output", doc="ffmpeg video codec for saved videos.") +FASTVIDEO_NVENC_PRESET = EnvStr("p1", category="output", doc="NVENC preset when the codec is an *_nvenc codec.") +FASTVIDEO_NVENC_TUNE = EnvStr("ull", category="output", doc="NVENC tune option.") +FASTVIDEO_NVENC_RC = EnvStr("constqp", category="output", doc="NVENC rate-control mode.") +FASTVIDEO_NVENC_QP = EnvStr("28", category="output", doc="NVENC quantization parameter.") +FASTVIDEO_NVENC_BF = EnvStr("0", category="output", doc="NVENC number of B-frames.") +FASTVIDEO_X264_PRESET = EnvStr("ultrafast", category="output", doc="x264 preset for non-NVENC codecs.") +FASTVIDEO_OUTPUT_PIX_FMT = EnvStr("yuv420p", category="output", doc="ffmpeg pixel format for saved videos.") # ================== Profiling ================== @@ -341,10 +419,6 @@ FASTVIDEO_TORCH_PROFILE_REGIONS = EnvStr( # ================== Debug ================== -FASTVIDEO_SERVER_DEV_MODE = EnvBool(False, - category="debug", - doc="Run the server in development mode with extra debugging endpoints.") -FASTVIDEO_TRACE_FUNCTION = EnvBool(False, category="debug", doc="Trace function calls.") FASTVIDEO_TRACE_ACTIVATIONS = EnvBool(False, category="debug", doc="Enable activation trace hooks.") FASTVIDEO_TRACE_LAYERS = EnvStr("", category="debug", doc="Regex filter for traced module names. Empty means all.") FASTVIDEO_TRACE_STATS = EnvStr("abs_mean,sum", @@ -354,6 +428,18 @@ FASTVIDEO_TRACE_OUTPUT = EnvStr("/tmp/fv_trace_.jsonl", category="debug", doc="JSONL path for activation traces. The literal is replaced at runtime.") FASTVIDEO_TRACE_STEPS = EnvStr("", category="debug", doc="Comma-separated denoising step indices. Empty means all.") +FASTVIDEO_H3_VSA_PROBE = EnvStr( + None, + category="debug", + doc="Output directory for the VSA-H3 attention-mass probe, which writes one .pt file per step, layer, and rank. " + "Keeps the model out of regional compile.") +FASTVIDEO_LTX2_GEMMA_LOG = EnvStr("", + category="debug", + doc="Log file for LTX-2 Gemma text-encoder hidden states, used by parity tests.", + deprecated_names=("LTX2_FASTVIDEO_GEMMA_LOG", )) +FASTVIDEO_COSMOS25_LOG_KNOBS = EnvBool(False, + category="debug", + doc="Log the Cosmos 2.5 latent-preparation conditioning inputs.") # ================== Sampling ================== @@ -385,6 +471,84 @@ FASTVIDEO_CFG_GATE_STEP = EnvFloat( category="sampling", doc="CFG gating fraction in [0, 1]. Steps before len(timesteps) * X run the conditional and unconditional " "forwards; later steps reuse the cached difference. 1.0 disables gating.") +FASTVIDEO_LTX2_USE_DISTILLED_SIGMAS = EnvBool( + True, + category="sampling", + doc="LTX-2 uses the distilled sigma schedule when FastVideoArgs.ltx2_use_distilled_sigmas is also true.", + deprecated_names=("LTX2_USE_DISTILLED_SIGMAS", )) + +# ================== Evaluation ================== + +FASTVIDEO_EVAL_CACHE = EnvPath(lambda: os.path.join(FASTVIDEO_CACHE_ROOT.get(), "eval"), + category="eval", + doc="Cache directory for evaluation models and datasets. Defaults to " + "$FASTVIDEO_CACHE_ROOT/eval.") +FASTVIDEO_PHYSICS_IQ_BUCKET_URL = EnvStr("https://storage.googleapis.com/physics-iq-benchmark", + category="eval", + doc="Base URL of the Physics-IQ benchmark bucket.") +FASTVIDEO_VBENCH_FULL_INFO_JSON = EnvStr(None, + category="eval", + doc="Path to VBench_full_info.json, used instead of the vendored copy.", + deprecated_names=("VBENCH_FULL_INFO_JSON", )) +FASTVIDEO_FVD_REF_FEATURES = EnvStr(None, category="eval", doc="Cached reference-feature file for the FVD metric.") +FASTVIDEO_FAD_REF_FEATURES = EnvStr(None, + category="eval", + doc="Cached reference-feature file for the audio Frechet distance metric.") + +# ================== Tests ================== + +FASTVIDEO_TEST_LTX2_OVERFIT_DATA_DIR = EnvStr("data/cats", + category="test", + doc="Raw data directory for preprocess_ltx2_overfit.py.", + deprecated_names=("LTX2_OVERFIT_DATA_DIR", )) +FASTVIDEO_TEST_LTX2_OVERFIT_CAPTION_JSON = EnvStr("videos2caption_1_sample.json", + category="test", + doc="Caption file, relative to the raw data directory.", + deprecated_names=("LTX2_OVERFIT_CAPTION_JSON", )) +FASTVIDEO_TEST_LTX2_OVERFIT_VIDEO_SUBDIR = EnvStr("video", + category="test", + doc="Video subdirectory, relative to the raw data directory.", + deprecated_names=("LTX2_OVERFIT_VIDEO_SUBDIR", )) +FASTVIDEO_TEST_LTX2_OVERFIT_OUTPUT_DIR = EnvStr("data/ltx2_overfit_preprocessed", + category="test", + doc="Output directory for preprocess_ltx2_overfit.py.", + deprecated_names=("LTX2_OVERFIT_OUTPUT_DIR", )) +FASTVIDEO_TEST_LTX2_OVERFIT_MODEL = EnvStr("FastVideo/LTX2-Distilled-Diffusers", + category="test", + doc="Model repository whose encoders preprocess_ltx2_overfit.py uses.", + deprecated_names=("LTX2_OVERFIT_MODEL", )) +FASTVIDEO_TEST_LTX2_OVERFIT_NUM_COPIES = EnvInt(4, + category="test", + doc="Number of copies of the overfit sample in the parquet file.", + deprecated_names=("LTX2_OVERFIT_NUM_COPIES", )) +FASTVIDEO_TEST_KANDINSKY5_OVERFIT_DATA_DIR = EnvStr("data/kandinsky5_overfit", + category="test", + doc="Raw data directory for preprocess_kandinsky5_overfit.py.", + deprecated_names=("KANDINSKY5_OVERFIT_DATA_DIR", )) +FASTVIDEO_TEST_KANDINSKY5_OVERFIT_OUTPUT_DIR = EnvStr("data/kandinsky5_overfit_preprocessed", + category="test", + doc="Output directory for preprocess_kandinsky5_overfit.py.", + deprecated_names=("KANDINSKY5_OVERFIT_OUTPUT_DIR", )) + +# Variables that FastVideo no longer reads. Setting one logs a warning; delete +# the entries in the next minor release. +DEPRECATED_VARIABLES = { + "FASTVIDEO_TARGET_DEVICE": "no code reads it", + "FASTVIDEO_USE_PRECOMPILED": "no code reads it", + "FASTVIDEO_RINGBUFFER_WARNING_INTERVAL": "no code reads it", + "FASTVIDEO_ENGINE_ITERATION_TIMEOUT_S": "no code reads it", + "FASTVIDEO_SERVER_DEV_MODE": "no code reads it", + "FASTVIDEO_TEST_DYNAMO_FULLGRAPH_CAPTURE": "no code reads it", + "FASTVIDEO_TRACE_FUNCTION": "no code reads it", +} + + +def warn_deprecated_variables() -> None: + """Log a warning for each variable in DEPRECATED_VARIABLES that is set.""" + for name, reason in DEPRECATED_VARIABLES.items(): + if name in os.environ: + _warn_once(f"{name} is deprecated and has no effect ({reason}); it will be removed in the next minor " + "release.") def _register_fields() -> dict[str, EnvField]: diff --git a/fastvideo/eval/__init__.py b/fastvideo/eval/__init__.py index 17decfcfe..6c989bf04 100644 --- a/fastvideo/eval/__init__.py +++ b/fastvideo/eval/__init__.py @@ -1,3 +1,4 @@ +import fastvideo.envs as envs from fastvideo.eval.models import ensure_checkpoint, get_cache_dir @@ -14,9 +15,8 @@ def _redirect_third_party_caches() -> None: env vars at all; their callsites in metric.py files pass ``download_root=str(get_cache_dir() / "")`` directly. """ - import os root = get_cache_dir() - os.environ.setdefault("TORCH_HOME", str(root / "torch")) + envs.setdefault_external("TORCH_HOME", str(root / "torch")) _redirect_third_party_caches() diff --git a/fastvideo/eval/datasets/physics_iq.py b/fastvideo/eval/datasets/physics_iq.py index f5ca10619..a90314685 100644 --- a/fastvideo/eval/datasets/physics_iq.py +++ b/fastvideo/eval/datasets/physics_iq.py @@ -27,7 +27,6 @@ pre-downloaded copy) to opt out of network fetches. from __future__ import annotations import csv -import os from dataclasses import dataclass from pathlib import Path from urllib.request import urlretrieve @@ -35,6 +34,7 @@ from urllib.request import urlretrieve import cv2 import numpy as np +import fastvideo.envs as envs from fastvideo.eval.datasets.base import PromptDataset from fastvideo.eval.datasets.registry import register_dataset from fastvideo.eval.models import get_cache_dir @@ -59,11 +59,10 @@ _VENDORED_DESCRIPTIONS_CSV = (Path(__file__).resolve().parent.parent / "metrics" # Public DeepMind bucket; HTTPS-readable, no auth. Override via # ``FASTVIDEO_PHYSICS_IQ_BUCKET_URL`` (e.g. for an internal mirror). -_DEFAULT_BUCKET_URL = "https://storage.googleapis.com/physics-iq-benchmark" def _bucket_url() -> str: - return os.environ.get("FASTVIDEO_PHYSICS_IQ_BUCKET_URL", _DEFAULT_BUCKET_URL) + return envs.FASTVIDEO_PHYSICS_IQ_BUCKET_URL.get() def _default_dataset_root() -> Path: diff --git a/fastvideo/eval/datasets/vbench.py b/fastvideo/eval/datasets/vbench.py index 61a08b025..9fbb93ff1 100644 --- a/fastvideo/eval/datasets/vbench.py +++ b/fastvideo/eval/datasets/vbench.py @@ -7,9 +7,9 @@ keyed by dimension). from __future__ import annotations import json -import os from pathlib import Path +import fastvideo.envs as envs from fastvideo.eval.datasets.base import PromptDataset from fastvideo.eval.datasets.registry import register_dataset @@ -23,12 +23,12 @@ _FULL_INFO_REL = "fastvideo/third_party/eval/vbench/vbench/VBench_full_info.json def _locate_full_info() -> Path: - env = os.environ.get("VBENCH_FULL_INFO_JSON") + env = envs.FASTVIDEO_VBENCH_FULL_INFO_JSON.get() if env: p = Path(env) if p.is_file(): return p - raise FileNotFoundError(f"VBENCH_FULL_INFO_JSON={env} does not point at a file") + raise FileNotFoundError(f"FASTVIDEO_VBENCH_FULL_INFO_JSON={env} does not point at a file") here = Path(__file__).resolve() for ancestor in here.parents: candidate = ancestor / _FULL_INFO_REL @@ -38,7 +38,7 @@ def _locate_full_info() -> Path: break raise FileNotFoundError("Could not locate VBench_full_info.json. Initialize the upstream " "submodule (`git submodule update --init " - "fastvideo/third_party/eval/vbench`) or set VBENCH_FULL_INFO_JSON.") + "fastvideo/third_party/eval/vbench`) or set FASTVIDEO_VBENCH_FULL_INFO_JSON.") @register_dataset("vbench") diff --git a/fastvideo/eval/metrics/audio/frechet_distance/metric.py b/fastvideo/eval/metrics/audio/frechet_distance/metric.py index f8fc3de58..aa091dd7d 100644 --- a/fastvideo/eval/metrics/audio/frechet_distance/metric.py +++ b/fastvideo/eval/metrics/audio/frechet_distance/metric.py @@ -11,7 +11,6 @@ two finite embeddings. from __future__ import annotations -import os from pathlib import Path from typing import Any @@ -19,6 +18,7 @@ import numpy as np import torch from scipy import linalg +import fastvideo.envs as envs from fastvideo.eval.metrics.base import BaseMetric from fastvideo.eval.registry import register from fastvideo.eval.types import MetricResult @@ -94,7 +94,7 @@ class FrechetAudioDistanceMetric(BaseMetric): self._model: Any = None self._gen_buf: list[np.ndarray] = [] self._ref_buf: list[np.ndarray] = [] - self._cached_ref_path: str | None = os.environ.get(REF_FEATURES_ENV) + self._cached_ref_path: str | None = envs.FASTVIDEO_FAD_REF_FEATURES.get() self._cached_ref_mu: np.ndarray | None = None self._cached_ref_sigma: np.ndarray | None = None self._n_cached_ref: int = 0 diff --git a/fastvideo/eval/metrics/common/fvd/metric.py b/fastvideo/eval/metrics/common/fvd/metric.py index 42536c086..e32e3aede 100644 --- a/fastvideo/eval/metrics/common/fvd/metric.py +++ b/fastvideo/eval/metrics/common/fvd/metric.py @@ -75,13 +75,13 @@ import numpy as np import scipy.linalg import torch +import fastvideo.envs as envs from fastvideo.eval.metrics.base import BaseMetric from fastvideo.eval.metrics.common.fvd.extractors import (_BaseExtractor, available_extractors, load_extractor) from fastvideo.eval.registry import register from fastvideo.eval.types import MetricResult _MIN_VIDEOS_WARN = 256 # below this FVD is unreliable -_REF_FEATURES_ENV = "FASTVIDEO_FVD_REF_FEATURES" def _gaussian_params(features: np.ndarray) -> tuple[np.ndarray, np.ndarray]: @@ -220,7 +220,7 @@ class FVDMetric(BaseMetric): # import. Precedence: kwarg > env-var > default. if self._cache_path_arg is not None: self.cache_path = os.path.expanduser(self._cache_path_arg) - elif env_path := os.environ.get(_REF_FEATURES_ENV): + elif env_path := envs.FASTVIDEO_FVD_REF_FEATURES.get(): self.cache_path = os.path.expanduser(env_path) else: self.cache_path = _default_cache_path(self._extractor_name) diff --git a/fastvideo/eval/metrics/judge/third_person_separation/metric.py b/fastvideo/eval/metrics/judge/third_person_separation/metric.py index d2feee2da..4692789b2 100644 --- a/fastvideo/eval/metrics/judge/third_person_separation/metric.py +++ b/fastvideo/eval/metrics/judge/third_person_separation/metric.py @@ -110,10 +110,9 @@ def _path_of(sample: dict, key: str) -> str | None: def _resolve_api_key() -> str: - for env in ("GEMINI_API_KEY", "GOOGLE_API_KEY"): - key = os.environ.get(env) - if key: - return key.strip() + key = os.environ.get("GEMINI_API_KEY") or os.environ.get("GOOGLE_API_KEY") + if key: + return key.strip() token = Path("~/.gemini_token").expanduser() if token.is_file(): return token.read_text().strip() diff --git a/fastvideo/eval/metrics/vbench/scene/metric.py b/fastvideo/eval/metrics/vbench/scene/metric.py index 77c3fe4ab..615cf63c2 100644 --- a/fastvideo/eval/metrics/vbench/scene/metric.py +++ b/fastvideo/eval/metrics/vbench/scene/metric.py @@ -21,6 +21,7 @@ import tempfile import torch import torchvision.io +import fastvideo.envs as envs from fastvideo.eval.metrics.base import BaseMetric from fastvideo.eval.registry import register from fastvideo.eval.types import MetricResult @@ -58,7 +59,7 @@ class SceneMetric(BaseMetric): from transformers import Qwen2_5OmniForConditionalGeneration, Qwen2_5OmniProcessor # AVoCaDO uses Qwen2.5-Omni — large multimodal model - os.environ.setdefault("VIDEO_MAX_PIXELS", str(20070400)) # 512*28*28*50 + envs.setdefault_external("VIDEO_MAX_PIXELS", str(20070400)) # 512*28*28*50 self._model = Qwen2_5OmniForConditionalGeneration.from_pretrained( self._model_path, diff --git a/fastvideo/eval/models.py b/fastvideo/eval/models.py index 9adb57b3e..339df2270 100644 --- a/fastvideo/eval/models.py +++ b/fastvideo/eval/models.py @@ -44,10 +44,7 @@ def get_cache_dir() -> Path: ``cache_dir``, etc.), pass ``str(get_cache_dir() / "")`` so users get a single ``FASTVIDEO_EVAL_CACHE`` knob to redirect them all. """ - return Path(os.environ.get( - "FASTVIDEO_EVAL_CACHE", - os.path.join(envs.FASTVIDEO_CACHE_ROOT.get(), "eval"), - )) + return Path(envs.FASTVIDEO_EVAL_CACHE.get()) def ensure_checkpoint( diff --git a/fastvideo/fastvideo_args.py b/fastvideo/fastvideo_args.py index 8cc5a60bf..8cb4ef920 100644 --- a/fastvideo/fastvideo_args.py +++ b/fastvideo/fastvideo_args.py @@ -361,6 +361,8 @@ class FastVideoArgs: if env_backend is not None and backend_name_to_enum(env_backend) is not None: self.attention_backend = env_backend self._fold_vae_parallel_env() + import fastvideo.envs as envs + envs.warn_deprecated_variables() self.check_fastvideo_args() def _fold_vae_parallel_env(self) -> None: diff --git a/fastvideo/logger.py b/fastvideo/logger.py index a930ae0e4..859c1f0df 100644 --- a/fastvideo/logger.py +++ b/fastvideo/logger.py @@ -15,11 +15,6 @@ from typing import Any, cast import fastvideo.envs as envs -FASTVIDEO_CONFIGURE_LOGGING = envs.FASTVIDEO_CONFIGURE_LOGGING.get() -FASTVIDEO_LOGGING_CONFIG_PATH = envs.FASTVIDEO_LOGGING_CONFIG_PATH.get() -FASTVIDEO_LOGGING_LEVEL = envs.FASTVIDEO_LOGGING_LEVEL.get() -FASTVIDEO_LOGGING_PREFIX = envs.FASTVIDEO_LOGGING_PREFIX.get() - RED = '\033[91m' GREEN = '\033[92m' RESET = '\033[0;0m' @@ -27,40 +22,43 @@ RESET = '\033[0;0m' _warned_local_main_process = False _warned_main_process = False -_FORMAT = (f"{FASTVIDEO_LOGGING_PREFIX}%(levelname)s %(asctime)s.%(msecs)03d " +_FORMAT = ("%(levelname)s %(asctime)s.%(msecs)03d " "[%(filename)s:%(lineno)d] %(message)s") _DATE_FORMAT = "%m-%d %H:%M:%S" -DEFAULT_LOGGING_CONFIG = { - "formatters": { - "fastvideo": { - "class": "fastvideo.logging_utils.NewLineFormatter", - "datefmt": _DATE_FORMAT, - "format": _FORMAT, + +def _default_logging_config() -> dict[str, Any]: + """Build the default logging configuration from FASTVIDEO_LOGGING_PREFIX and FASTVIDEO_LOGGING_LEVEL.""" + return { + "formatters": { + "fastvideo": { + "class": "fastvideo.logging_utils.NewLineFormatter", + "datefmt": _DATE_FORMAT, + "format": envs.FASTVIDEO_LOGGING_PREFIX.get() + _FORMAT, + }, }, - }, - "handlers": { - "fastvideo": { - "class": "logging.StreamHandler", - "formatter": "fastvideo", - "level": FASTVIDEO_LOGGING_LEVEL, - "stream": "ext://sys.stdout", + "handlers": { + "fastvideo": { + "class": "logging.StreamHandler", + "formatter": "fastvideo", + "level": envs.FASTVIDEO_LOGGING_LEVEL.get(), + "stream": "ext://sys.stdout", + }, }, - }, - "loggers": { - "fastvideo": { + "loggers": { + "fastvideo": { + "handlers": ["fastvideo"], + "level": "DEBUG", + "propagate": False, + }, + }, + "root": { "handlers": ["fastvideo"], "level": "DEBUG", - "propagate": False, }, - }, - "root": { - "handlers": ["fastvideo"], - "level": "DEBUG", - }, - "version": 1, - "disable_existing_loggers": False -} + "version": 1, + "disable_existing_loggers": False + } @lru_cache @@ -180,20 +178,22 @@ class _FastvideoLogger(Logger): def _configure_fastvideo_root_logger() -> None: logging_config = dict[str, Any]() + configure_logging = envs.FASTVIDEO_CONFIGURE_LOGGING.get() + logging_config_path = envs.FASTVIDEO_LOGGING_CONFIG_PATH.get() - if not FASTVIDEO_CONFIGURE_LOGGING and FASTVIDEO_LOGGING_CONFIG_PATH: + if not configure_logging and logging_config_path: raise RuntimeError("FASTVIDEO_CONFIGURE_LOGGING evaluated to false, but " "FASTVIDEO_LOGGING_CONFIG_PATH was given. FASTVIDEO_LOGGING_CONFIG_PATH " "implies FASTVIDEO_CONFIGURE_LOGGING. Please enable " "FASTVIDEO_CONFIGURE_LOGGING or unset FASTVIDEO_LOGGING_CONFIG_PATH.") - if FASTVIDEO_CONFIGURE_LOGGING: - logging_config = DEFAULT_LOGGING_CONFIG + if configure_logging: + logging_config = _default_logging_config() - if FASTVIDEO_LOGGING_CONFIG_PATH: - if not path.exists(FASTVIDEO_LOGGING_CONFIG_PATH): - raise RuntimeError("Could not load logging config. File does not exist: %s", FASTVIDEO_LOGGING_CONFIG_PATH) - with open(FASTVIDEO_LOGGING_CONFIG_PATH, encoding="utf-8") as file: + if logging_config_path: + if not path.exists(logging_config_path): + raise RuntimeError("Could not load logging config. File does not exist: %s", logging_config_path) + with open(logging_config_path, encoding="utf-8") as file: custom_config = json.loads(file.read()) if not isinstance(custom_config, dict): @@ -286,7 +286,7 @@ def enable_trace_function_call(log_file_path: str, root_dir: str | None = None): Note that this call is thread-level, any threads calling this function will have the trace enabled. Other threads will not be affected. """ - logger.warning("FASTVIDEO_TRACE_FUNCTION is enabled. It will record every" + logger.warning("Function-call tracing is enabled. It will record every" " function executed by Python. This will slow down the code. It " "is suggested to be used for debugging hang or crashes only.") logger.info("Trace frame log is saved to %s", log_file_path) diff --git a/fastvideo/mlx_runtime/fastwan.py b/fastvideo/mlx_runtime/fastwan.py index affa6e389..d6a802b2b 100644 --- a/fastvideo/mlx_runtime/fastwan.py +++ b/fastvideo/mlx_runtime/fastwan.py @@ -6,7 +6,6 @@ from __future__ import annotations import json import math -import os import statistics import time from dataclasses import dataclass @@ -14,6 +13,7 @@ from pathlib import Path from typing import TYPE_CHECKING, Any from collections.abc import Callable +import fastvideo.envs as envs from fastvideo.logger import init_logger if TYPE_CHECKING: @@ -326,7 +326,7 @@ def dq_gemm_engaged() -> int: def affine_dq_gemm_min_m() -> int | None: - raw = os.environ.get("FASTVIDEO_MLX_DQ_GEMM", "1").strip().lower() + raw = envs.FASTVIDEO_MLX_DQ_GEMM.get().strip().lower() if raw in {"", "0", "off", "false", "no"}: return None if raw in {"1", "on", "true", "yes"}: @@ -400,9 +400,7 @@ def _use_fast_norm() -> bool: traffic) and benchmark the speedup. H3 uses its own fused RMSNorm default and does not consult this toggle. """ - import os - - return os.environ.get("FASTVIDEO_MLX_FAST_NORM", "0") == "1" + return envs.FASTVIDEO_MLX_FAST_NORM.get() def layer_norm(x, weight=None, bias=None, eps: float = 1e-6): @@ -633,12 +631,12 @@ class MLXWanTransformerBlock: k_bh = key.transpose(0, 2, 1, 3) v_bh = value.transpose(0, 2, 1, 3) scale = self.head_dim**-0.5 - window = int(os.environ.get("FASTVIDEO_MLX_WINDOW", "0") or "0") + window = envs.FASTVIDEO_MLX_WINDOW.get() if window > 0: from fastvideo.mlx_runtime.windowed_attention import windowed_attention _warn_windowed_attention_once(window) - sink = int(os.environ.get("FASTVIDEO_MLX_WINDOW_SINK", "0") or "0") + sink = envs.FASTVIDEO_MLX_WINDOW_SINK.get() attn_output = windowed_attention(q_bh, k_bh, v_bh, window=window, sink=sink, scale=scale) else: attn_output = mx.fast.scaled_dot_product_attention(q_bh, k_bh, v_bh, scale=scale) @@ -693,7 +691,6 @@ class MLXWanDiT: *, compile: bool = False, ) -> None: - import os self.weights = weights self.blocks = blocks @@ -711,7 +708,7 @@ class MLXWanDiT: # good mx.compile target. Off by default so the eager path stays the # baseline; enable via constructor or FASTVIDEO_MLX_COMPILE=1 and verify # with the benchmark's SSIM ~= 1.0 check. - self._enable_compile = compile or os.environ.get("FASTVIDEO_MLX_COMPILE", "0") == "1" + self._enable_compile = compile or envs.FASTVIDEO_MLX_COMPILE.get() self._compiled_forward: Callable[..., Any] | None = None self._compiled_signature: tuple | None = None diff --git a/fastvideo/mlx_runtime/wan22.py b/fastvideo/mlx_runtime/wan22.py index 6171481a6..2fe0e59c7 100644 --- a/fastvideo/mlx_runtime/wan22.py +++ b/fastvideo/mlx_runtime/wan22.py @@ -23,6 +23,7 @@ from pathlib import Path from typing import TYPE_CHECKING, Any from collections.abc import Callable +import fastvideo.envs as envs from fastvideo.logger import init_logger from fastvideo.mlx_runtime.fastwan import ( MLXWanT2VCrossAttention, @@ -132,7 +133,6 @@ class MLXWan22DiT: *, compile: bool = False, ) -> None: - import os self.weights = weights self.blocks = blocks @@ -144,7 +144,7 @@ class MLXWan22DiT: self.patch_size = tuple(config["patch_size"]) self.out_channels = int(config["out_channels"]) self.eps = float(config.get("eps", 1e-6)) - self._enable_compile = compile or os.environ.get("FASTVIDEO_MLX_COMPILE", "0") == "1" + self._enable_compile = compile or envs.FASTVIDEO_MLX_COMPILE.get() self._compiled_forward: Callable[..., Any] | None = None self._compiled_signature: tuple | None = None diff --git a/fastvideo/models/encoders/gemma.py b/fastvideo/models/encoders/gemma.py index 76c440f13..5b36d1004 100644 --- a/fastvideo/models/encoders/gemma.py +++ b/fastvideo/models/encoders/gemma.py @@ -9,6 +9,7 @@ import torch from torch import nn from transformers import AutoTokenizer, Gemma3ForConditionalGeneration +import fastvideo.envs as envs from fastvideo.configs.models.encoders import BaseEncoderOutput, TextEncoderConfig from fastvideo.models.encoders.base import TextEncoder from fastvideo.models.dits.ltx2 import ( @@ -27,7 +28,7 @@ import math def _debug_gemma_log_line(message: str) -> None: - log_path = os.getenv("LTX2_FASTVIDEO_GEMMA_LOG", "") + log_path = envs.FASTVIDEO_LTX2_GEMMA_LOG.get() if not log_path: return log_dir = os.path.dirname(log_path) @@ -459,7 +460,7 @@ class LTX2GemmaTextEncoderModel(TextEncoder): padding_side: str, ) -> tuple[torch.Tensor, torch.Tensor]: encoded_text_features = torch.stack(hidden_states, dim=-1) - if os.getenv("LTX2_FASTVIDEO_GEMMA_LOG", ""): + if envs.FASTVIDEO_LTX2_GEMMA_LOG.get(): for idx, layer in enumerate(hidden_states): _debug_gemma_log_line(f"fastvideo:gemma_hidden_state_{idx}" f":sum={layer.float().sum().item():.6f}") diff --git a/fastvideo/models/encoders/reason1.py b/fastvideo/models/encoders/reason1.py index 51f9f711c..1784b26e7 100644 --- a/fastvideo/models/encoders/reason1.py +++ b/fastvideo/models/encoders/reason1.py @@ -1,13 +1,13 @@ # SPDX-License-Identifier: Apache-2.0 """Reason1 (Qwen2.5-VL) text encoder.""" -import os from dataclasses import dataclass from collections.abc import Iterable import torch from transformers import AutoProcessor +import fastvideo.envs as envs from fastvideo.configs.models.encoders import BaseEncoderOutput, Reason1Config from fastvideo.logger import init_logger from fastvideo.models.encoders.base import TextEncoder @@ -89,7 +89,7 @@ class Reason1TextEncoder(TextEncoder): trust_remote_code=True, ) - weights_override = os.getenv("FASTVIDEO_REASON1_WEIGHTS_PATH") + weights_override = envs.FASTVIDEO_REASON1_WEIGHTS_PATH.get() if weights_override: self.secondary_weights = (_WeightsSource( model_or_path=weights_override, diff --git a/fastvideo/models/loader/benchmarks/benchmark_weight_loading.py b/fastvideo/models/loader/benchmarks/benchmark_weight_loading.py index 0ce6d64b8..6c5995c63 100644 --- a/fastvideo/models/loader/benchmarks/benchmark_weight_loading.py +++ b/fastvideo/models/loader/benchmarks/benchmark_weight_loading.py @@ -21,6 +21,7 @@ import time import torch import torch.distributed as dist +import fastvideo.envs as envs from fastvideo.distributed.parallel_state import ( cleanup_dist_env_and_memory, get_node_group, @@ -121,14 +122,10 @@ def main() -> None: files = resolve_safetensors_files(load_path) # Set default env vars for single-GPU standalone mode (no torchrun) - if "MASTER_ADDR" not in os.environ: - os.environ["MASTER_ADDR"] = "localhost" - if "MASTER_PORT" not in os.environ: - os.environ["MASTER_PORT"] = "29500" - if "RANK" not in os.environ: - os.environ["RANK"] = "0" - if "WORLD_SIZE" not in os.environ: - os.environ["WORLD_SIZE"] = "1" + envs.setdefault_external("MASTER_ADDR", "localhost") + envs.setdefault_external("MASTER_PORT", "29500") + envs.setdefault_external("RANK", "0") + envs.setdefault_external("WORLD_SIZE", "1") maybe_init_distributed_environment_and_model_parallel(1, 1) rank = dist.get_rank() if dist.is_initialized() else 0 diff --git a/fastvideo/models/loader/fsdp_load.py b/fastvideo/models/loader/fsdp_load.py index f7c31d221..956850a50 100644 --- a/fastvideo/models/loader/fsdp_load.py +++ b/fastvideo/models/loader/fsdp_load.py @@ -5,7 +5,6 @@ # Copyright 2025 The FastVideo Authors. from __future__ import annotations -import os import contextlib import re from collections.abc import Callable, Generator @@ -392,7 +391,7 @@ def _regional_compile_unsupported_reason( resolved = getattr(config, "_resolved_attention_backend", None) resolved_name = getattr(resolved, "name", "") if resolved_name == "VIDEO_SPARSE_ATTN_H3": - if os.environ.get("FASTVIDEO_H3_VSA_PROBE"): + if envs.FASTVIDEO_H3_VSA_PROBE.get(): return ("FASTVIDEO_H3_VSA_PROBE records tensors and files from the VSA-H3 attention body, which " "regional fullgraph compile cannot capture; this model stays eager") if not envs.FASTVIDEO_VSA_SM100A.get(): @@ -508,7 +507,7 @@ def shard_model( ValueError: If no layer modules were sharded, indicating that no shard_condition was triggered. """ # Check if we should use size-based filtering - use_size_filtering = os.environ.get("FASTVIDEO_FSDP2_AUTOWRAP", "0") == "1" + use_size_filtering = envs.FASTVIDEO_FSDP2_AUTOWRAP.get() if not fsdp_shard_conditions: logger.warning("No FSDP shard conditions provided; nothing will be sharded.") @@ -543,7 +542,7 @@ def shard_model( if use_size_filtering: # Size-based filtering mode - min_params = int(os.environ.get("FASTVIDEO_FSDP2_MIN_PARAMS", "10000000")) + min_params = envs.FASTVIDEO_FSDP2_MIN_PARAMS.get() logger.info("Using size-based filtering with threshold: %.2fM", min_params / 1e6) for n, m in reversed(named_modules): diff --git a/fastvideo/models/vaes/ltx2vae.py b/fastvideo/models/vaes/ltx2vae.py index 8789d352a..8556c1bea 100644 --- a/fastvideo/models/vaes/ltx2vae.py +++ b/fastvideo/models/vaes/ltx2vae.py @@ -7,7 +7,6 @@ from __future__ import annotations import itertools import math -import os from dataclasses import dataclass, replace from enum import Enum from typing import Any, Callable, Iterator, List, NamedTuple, Tuple @@ -17,14 +16,10 @@ import torch.nn as nn import torch.nn.functional as F from einops import rearrange +import fastvideo.envs as envs from fastvideo.models.vaes.common import DiagonalGaussianDistribution -def _is_env_enabled(name: str, default: str = "") -> bool: - value = os.getenv(name, default) - return value.lower() in {"1", "true", "yes", "on"} - - # ============================================================================= # Enums # ============================================================================= @@ -1595,7 +1590,7 @@ class LTX2CausalVideoAutoencoder(nn.Module): self._use_tiling: bool = False self._use_channels_last_3d: bool = False - if _is_env_enabled("FASTVIDEO_LTX2_VAE_CHANNELS_LAST_3D", default="1"): + if envs.FASTVIDEO_LTX2_VAE_CHANNELS_LAST_3D.get(): self.enable_channels_last_3d() def _as_channels_last_3d(self, tensor: torch.Tensor) -> torch.Tensor: diff --git a/fastvideo/pipelines/basic/ltx2/stages/ltx2_audio_decoding.py b/fastvideo/pipelines/basic/ltx2/stages/ltx2_audio_decoding.py index e191eec2b..e868e0010 100644 --- a/fastvideo/pipelines/basic/ltx2/stages/ltx2_audio_decoding.py +++ b/fastvideo/pipelines/basic/ltx2/stages/ltx2_audio_decoding.py @@ -5,10 +5,9 @@ Audio decoding stage for LTX-2 pipelines. from __future__ import annotations -import os - import torch +import fastvideo.envs as envs from fastvideo.distributed import get_local_torch_device from fastvideo.fastvideo_args import FastVideoArgs from fastvideo.logger import init_logger @@ -55,7 +54,7 @@ class LTX2AudioDecodingStage(PipelineStage): self.vocoder = self.vocoder.to(device) audio_latents = audio_latents.to(device) - disable_autocast = os.getenv("LTX2_DISABLE_AUDIO_AUTOCAST", "1") == "1" + disable_autocast = envs.FASTVIDEO_LTX2_DISABLE_AUDIO_AUTOCAST.get() with torch.no_grad(), torch.autocast( device_type="cuda", dtype=audio_latents.dtype, diff --git a/fastvideo/pipelines/basic/ltx2/stages/ltx2_denoising.py b/fastvideo/pipelines/basic/ltx2/stages/ltx2_denoising.py index 4cd9aa412..4fc98e5a5 100644 --- a/fastvideo/pipelines/basic/ltx2/stages/ltx2_denoising.py +++ b/fastvideo/pipelines/basic/ltx2/stages/ltx2_denoising.py @@ -8,12 +8,12 @@ from __future__ import annotations from contextlib import contextmanager from itertools import combinations import math -import os from pathlib import Path import torch from tqdm.auto import tqdm +import fastvideo.envs as envs from fastvideo.attention.backends.video_sparse_attn import (VideoSparseAttentionMetadataBuilder) from fastvideo.attention.selector import component_attention_backend from fastvideo.fastvideo_args import FastVideoArgs @@ -54,7 +54,7 @@ except ImportError: @contextmanager def _nvtx_range(name: str): - if os.getenv("FASTVIDEO_NVTX_PROFILE", "0") == "1" and torch.cuda.is_available(): + if envs.FASTVIDEO_NVTX_PROFILE.get() and torch.cuda.is_available(): torch.cuda.nvtx.range_push(name) try: yield @@ -245,7 +245,7 @@ class LTX2DenoisingStage(PipelineStage): else: # Use distilled hardcoded schedule (or subsets) when enabled. use_distilled_sigmas = (fastvideo_args.ltx2_use_distilled_sigmas - and os.getenv("LTX2_USE_DISTILLED_SIGMAS", "1") == "1") + and envs.FASTVIDEO_LTX2_USE_DISTILLED_SIGMAS.get()) max_distilled_steps = len(DISTILLED_SIGMA_VALUES) - 1 if use_distilled_sigmas and num_inference_steps <= max_distilled_steps: sigmas, distilled_indices = _distilled_subset_sigmas( diff --git a/fastvideo/pipelines/basic/magi_human/magi_human_pipeline.py b/fastvideo/pipelines/basic/magi_human/magi_human_pipeline.py index bac774807..d7f5aa02f 100644 --- a/fastvideo/pipelines/basic/magi_human/magi_human_pipeline.py +++ b/fastvideo/pipelines/basic/magi_human/magi_human_pipeline.py @@ -29,6 +29,7 @@ from typing import Any from transformers import AutoTokenizer +import fastvideo.envs as envs from fastvideo.configs.models.encoders.t5gemma import T5GemmaEncoderConfig from fastvideo.configs.models.vaes import OobleckVAEConfig from fastvideo.fastvideo_args import FastVideoArgs @@ -71,7 +72,7 @@ def _ensure_hf_token_env() -> str | None: """ token = resolve_hf_token() if token: - os.environ.setdefault("HF_TOKEN", token) + envs.setdefault_external("HF_TOKEN", token) return token diff --git a/fastvideo/pipelines/preprocess/preprocess_kandinsky5_overfit.py b/fastvideo/pipelines/preprocess/preprocess_kandinsky5_overfit.py index 357a34fef..1186b48b2 100644 --- a/fastvideo/pipelines/preprocess/preprocess_kandinsky5_overfit.py +++ b/fastvideo/pipelines/preprocess/preprocess_kandinsky5_overfit.py @@ -26,7 +26,7 @@ Usage: Input/output roots default to ``data/kandinsky5_overfit`` / ``data/kandinsky5_overfit_preprocessed`` and can be overridden with the -``KANDINSKY5_OVERFIT_DATA_DIR`` / ``KANDINSKY5_OVERFIT_OUTPUT_DIR`` env +``FASTVIDEO_TEST_KANDINSKY5_OVERFIT_DATA_DIR`` / ``FASTVIDEO_TEST_KANDINSKY5_OVERFIT_OUTPUT_DIR`` env vars (used by the nightly e2e test to keep its disposable roots separate from a user's real dataset at the defaults). """ @@ -34,9 +34,6 @@ from a user's real dataset at the defaults). import json import os -os.environ.setdefault("MASTER_ADDR", "localhost") -os.environ.setdefault("MASTER_PORT", "29520") - import cv2 import numpy as np import pyarrow as pa @@ -44,6 +41,7 @@ import pyarrow.parquet as pq import torch from transformers import AutoTokenizer +import fastvideo.envs as envs from fastvideo.configs.pipelines.base import preprocess_text from fastvideo.configs.pipelines.kandinsky5 import ( Kandinsky5T2VConfig, @@ -70,11 +68,6 @@ MAX_WIDTH = 768 TRAIN_FPS = 24.0 MODEL_PATH = "kandinskylab/Kandinsky-5.0-T2V-Lite-sft-5s-Diffusers" -# Overridable so automation (e.g. the nightly e2e test) can point at its own -# test-owned directories instead of clobbering the documented default paths -# a user may have populated with their real dataset. -DATA_DIR = os.environ.get("KANDINSKY5_OVERFIT_DATA_DIR", "data/kandinsky5_overfit") -OUTPUT_DIR = os.environ.get("KANDINSKY5_OVERFIT_OUTPUT_DIR", "data/kandinsky5_overfit_preprocessed") def load_video(path: str, num_frames: int, height: int, width: int) -> torch.Tensor: @@ -143,17 +136,25 @@ def main() -> None: # FastVideo's native text-encoder layers (e.g. CLIPAttention's # QKVParallelLinear) are tensor-parallel-aware and assert the TP process # group is initialized, even for a single-process/single-GPU run. + envs.setdefault_external("MASTER_ADDR", "localhost") + envs.setdefault_external("MASTER_PORT", "29520") maybe_init_distributed_environment_and_model_parallel(1, 1) + # Overridable so automation (e.g. the nightly e2e test) can point at its own + # test-owned directories instead of clobbering the documented default paths + # a user may have populated with their real dataset. + data_dir = envs.FASTVIDEO_TEST_KANDINSKY5_OVERFIT_DATA_DIR.get() + output_dir = envs.FASTVIDEO_TEST_KANDINSKY5_OVERFIT_OUTPUT_DIR.get() + device = torch.device("cuda:0") model_path = maybe_download_model(MODEL_PATH) - os.makedirs(OUTPUT_DIR, exist_ok=True) + os.makedirs(output_dir, exist_ok=True) # Load captions. Validate up front (before spending minutes loading the # VAE + two text encoders): an empty/malformed manifest would otherwise # only surface as an IndexError on records[0] at parquet-write time. - manifest_path = os.path.join(DATA_DIR, "videos2caption.json") + manifest_path = os.path.join(data_dir, "videos2caption.json") with open(manifest_path) as f: caption_data = json.load(f) if not isinstance(caption_data, list) or not caption_data: @@ -212,7 +213,7 @@ def main() -> None: for item in caption_data: video_name = item["path"] caption = get_caption(item) - video_path = os.path.join(DATA_DIR, "videos", video_name) + video_path = os.path.join(data_dir, "videos", video_name) print(f"\nProcessing: {video_name}") print(f" Caption: {caption[:80]}...") @@ -289,7 +290,7 @@ def main() -> None: for k in records[0]}, schema=pyarrow_schema_t2v, ) - output_path = os.path.join(OUTPUT_DIR, "data_00000.parquet") + output_path = os.path.join(output_dir, "data_00000.parquet") pq.write_table(table, output_path) print(f"\nWrote {len(records)} records to {output_path}") @@ -297,12 +298,12 @@ def main() -> None: # Wrap in "data" key -- ValidationDataset expects field="data". # Use "caption" field -- ValidationDataset aliases it to "prompt". val_prompts = {"data": [{"caption": get_caption(item)} for item in caption_data]} - val_path = os.path.join(OUTPUT_DIR, "validation_prompts.json") + val_path = os.path.join(output_dir, "validation_prompts.json") with open(val_path, "w") as f: json.dump(val_prompts, f, indent=2) print(f"Wrote validation prompts to {val_path}") - print("\nDone! Use data_path: " + OUTPUT_DIR + " in training config.") + print("\nDone! Use data_path: " + output_dir + " in training config.") if __name__ == "__main__": diff --git a/fastvideo/pipelines/preprocess/preprocess_ltx2_overfit.py b/fastvideo/pipelines/preprocess/preprocess_ltx2_overfit.py index 2f55261b9..4eca9a29c 100644 --- a/fastvideo/pipelines/preprocess/preprocess_ltx2_overfit.py +++ b/fastvideo/pipelines/preprocess/preprocess_ltx2_overfit.py @@ -32,6 +32,7 @@ import pyarrow as pa import pyarrow.parquet as pq import torch +import fastvideo.envs as envs from fastvideo.dataset.dataloader.schema import pyarrow_schema_t2v from fastvideo.utils import maybe_download_model, verify_model_config_and_directory @@ -41,25 +42,14 @@ MAX_HEIGHT = 480 # divisible by 32 (spatial compression) MAX_WIDTH = 832 TRAIN_FPS = 24.0 # matches the LTX-2 preset fps used at validation -DATA_DIR = os.environ.get("LTX2_OVERFIT_DATA_DIR", "data/cats") -CAPTION_JSON = os.environ.get("LTX2_OVERFIT_CAPTION_JSON", "videos2caption_1_sample.json") -VIDEO_SUBDIR = os.environ.get("LTX2_OVERFIT_VIDEO_SUBDIR", "video") -OUTPUT_DIR = os.environ.get("LTX2_OVERFIT_OUTPUT_DIR", "data/ltx2_overfit_preprocessed") -MODEL_REPO = os.environ.get("LTX2_OVERFIT_MODEL", "FastVideo/LTX2-Distilled-Diffusers") -# The train dataloader samples with drop_last=True across data-parallel -# groups, so the dataset must hold at least num_sp_groups * batch_size -# rows or every rank gets zero batches. Replicate the overfit sample so -# a 4-GPU FSDP run still sees one batch per rank. -NUM_COPIES = int(os.environ.get("LTX2_OVERFIT_NUM_COPIES", "4")) - def _init_single_process_distributed() -> None: """FastVideo component loaders expect an initialized distributed env.""" - os.environ.setdefault("MASTER_ADDR", "127.0.0.1") - os.environ.setdefault("MASTER_PORT", "29511") - os.environ.setdefault("RANK", "0") - os.environ.setdefault("WORLD_SIZE", "1") - os.environ.setdefault("LOCAL_RANK", "0") + envs.setdefault_external("MASTER_ADDR", "127.0.0.1") + envs.setdefault_external("MASTER_PORT", "29511") + envs.setdefault_external("RANK", "0") + envs.setdefault_external("WORLD_SIZE", "1") + envs.setdefault_external("LOCAL_RANK", "0") from fastvideo.distributed import ( maybe_init_distributed_environment_and_model_parallel, ) maybe_init_distributed_environment_and_model_parallel(1, 1) @@ -106,16 +96,27 @@ def main() -> None: PipelineComponentLoader, ) from fastvideo.pipelines.basic.ltx2.pipeline_configs import LTX2T2VConfig + data_dir = envs.FASTVIDEO_TEST_LTX2_OVERFIT_DATA_DIR.get() + caption_json = envs.FASTVIDEO_TEST_LTX2_OVERFIT_CAPTION_JSON.get() + video_subdir = envs.FASTVIDEO_TEST_LTX2_OVERFIT_VIDEO_SUBDIR.get() + output_dir = envs.FASTVIDEO_TEST_LTX2_OVERFIT_OUTPUT_DIR.get() + model_repo = envs.FASTVIDEO_TEST_LTX2_OVERFIT_MODEL.get() + # The train dataloader samples with drop_last=True across data-parallel + # groups, so the dataset must hold at least num_sp_groups * batch_size + # rows or every rank gets zero batches. Replicate the overfit sample so + # a 4-GPU FSDP run still sees one batch per rank. + num_copies = envs.FASTVIDEO_TEST_LTX2_OVERFIT_NUM_COPIES.get() + device = torch.device("cuda:0") - model_path = maybe_download_model(MODEL_REPO) + model_path = maybe_download_model(model_repo) model_index = verify_model_config_and_directory(model_path) - os.makedirs(OUTPUT_DIR, exist_ok=True) + os.makedirs(output_dir, exist_ok=True) # The map-style dataset caches parquet file metadata; a stale cache # next to a regenerated parquet can crash or serve old rows. - shutil.rmtree(os.path.join(OUTPUT_DIR, "map_style_cache"), ignore_errors=True) + shutil.rmtree(os.path.join(output_dir, "map_style_cache"), ignore_errors=True) - with open(os.path.join(DATA_DIR, CAPTION_JSON)) as f: + with open(os.path.join(data_dir, caption_json)) as f: caption_data = json.load(f) pipeline_config = LTX2T2VConfig() @@ -163,7 +164,7 @@ def main() -> None: video_name = item["path"] record_id = f"{idx:04d}_{video_name}" caption = item["cap"][0] if isinstance(item["cap"], list) else item["cap"] - video_path = os.path.join(DATA_DIR, VIDEO_SUBDIR, video_name) + video_path = os.path.join(data_dir, video_subdir, video_name) print(f"\nProcessing: {video_name}") print(f" Caption: {caption[:80]}...") @@ -213,7 +214,7 @@ def main() -> None: # Save the preprocessed clip so overfit tests can compare # validation output against the memorization target. import imageio - ref_path = os.path.join(OUTPUT_DIR, f"training_sample_{idx}.mp4") + ref_path = os.path.join(output_dir, f"training_sample_{idx}.mp4") with imageio.get_writer(ref_path, fps=TRAIN_FPS) as writer: for frame in frames_np: writer.append_data(frame) @@ -223,9 +224,9 @@ def main() -> None: del text_encoder, tokenizer, vae torch.cuda.empty_cache() - # Write parquet (replicated NUM_COPIES times; see comment at top) + # Write parquet (replicated num_copies times; see the comment above num_copies) replicated = [] - for copy_idx in range(max(1, NUM_COPIES)): + for copy_idx in range(max(1, num_copies)): for r in records: row = dict(r) row["id"] = f"{r['id']}_copy{copy_idx}" @@ -235,10 +236,10 @@ def main() -> None: for k in replicated[0]}, schema=pyarrow_schema_t2v, ) - output_path = os.path.join(OUTPUT_DIR, "data_00000.parquet") + output_path = os.path.join(output_dir, "data_00000.parquet") pq.write_table(table, output_path) print(f"\nWrote {len(replicated)} records " - f"({len(records)} unique x {max(1, NUM_COPIES)} copies) to {output_path}") + f"({len(records)} unique x {max(1, num_copies)} copies) to {output_path}") # Write validation prompts for the validation callback val_prompts = { @@ -246,12 +247,12 @@ def main() -> None: "caption": (item["cap"][0] if isinstance(item["cap"], list) else item["cap"]), } for item in caption_data], } - val_path = os.path.join(OUTPUT_DIR, "validation_prompts.json") + val_path = os.path.join(output_dir, "validation_prompts.json") with open(val_path, "w") as f: json.dump(val_prompts, f, indent=2) print(f"Wrote validation prompts to {val_path}") - print("\nDone! Use data_path: " + OUTPUT_DIR + " in training config.") + print("\nDone! Use data_path: " + output_dir + " in training config.") if __name__ == "__main__": diff --git a/fastvideo/pipelines/preprocess/preprocess_minimax_h3_overfit.py b/fastvideo/pipelines/preprocess/preprocess_minimax_h3_overfit.py index 33a26ce99..9ef03db80 100644 --- a/fastvideo/pipelines/preprocess/preprocess_minimax_h3_overfit.py +++ b/fastvideo/pipelines/preprocess/preprocess_minimax_h3_overfit.py @@ -10,7 +10,6 @@ from __future__ import annotations import argparse import gc import json -import os from pathlib import Path import shutil from typing import Any @@ -20,6 +19,7 @@ import pyarrow as pa import pyarrow.parquet as pq import torch +import fastvideo.envs as envs from fastvideo.configs.pipelines.minimax_h3 import MiniMaxH3PipelineConfig from fastvideo.dataset.dataloader.schema import pyarrow_schema_t2va from fastvideo.fastvideo_args import FastVideoArgs @@ -50,11 +50,11 @@ TRAINING_VIDEO_NAME = "1gGQy4nxyUo-Scene-016.mp4" def _init_single_process_distributed() -> None: """Initialize the one-rank process groups required by component loaders.""" - os.environ.setdefault("MASTER_ADDR", "127.0.0.1") - os.environ.setdefault("MASTER_PORT", "29531") - os.environ.setdefault("RANK", "0") - os.environ.setdefault("WORLD_SIZE", "1") - os.environ.setdefault("LOCAL_RANK", "0") + envs.setdefault_external("MASTER_ADDR", "127.0.0.1") + envs.setdefault_external("MASTER_PORT", "29531") + envs.setdefault_external("RANK", "0") + envs.setdefault_external("WORLD_SIZE", "1") + envs.setdefault_external("LOCAL_RANK", "0") from fastvideo.distributed import maybe_init_distributed_environment_and_model_parallel maybe_init_distributed_environment_and_model_parallel(1, 1) diff --git a/fastvideo/pipelines/stages/denoising.py b/fastvideo/pipelines/stages/denoising.py index f36509eb3..5c867c969 100644 --- a/fastvideo/pipelines/stages/denoising.py +++ b/fastvideo/pipelines/stages/denoising.py @@ -4,7 +4,6 @@ Denoising stage for diffusion pipelines. """ import inspect -import os import weakref from collections.abc import Iterable from dataclasses import dataclass @@ -141,8 +140,7 @@ class DenoisingStage(PipelineStage): # gates should graduate to arch-config declarations like the precision # policies above. _is_flux = (getattr(fastvideo_args.pipeline_config.dit_config, "prefix", "") == "Flux") - if _is_flux and os.getenv("FASTVIDEO_FLUX2_DISABLE_BF16_REDUCED_PRECISION_REDUCTION", - "").lower() in {"1", "true", "yes"}: + if _is_flux and envs.FASTVIDEO_FLUX2_DISABLE_BF16_REDUCED_PRECISION_REDUCTION.get(): # Gate 1: tighten bf16 matmul accumulation for the 4-step Klein model (opt-in via env var). torch.backends.cuda.matmul.allow_bf16_reduced_precision_reduction = False # Gate 2: Flux2 runs its bf16 transformer WITHOUT autocast — autocast perturbs long-sequence attention enough to break 4-step latent parity. diff --git a/fastvideo/pipelines/stages/latent_preparation.py b/fastvideo/pipelines/stages/latent_preparation.py index 5f60409de..c01e6c7f8 100644 --- a/fastvideo/pipelines/stages/latent_preparation.py +++ b/fastvideo/pipelines/stages/latent_preparation.py @@ -3,13 +3,13 @@ Latent preparation stage for diffusion pipelines. """ -import os from typing import Any import numpy as np import torch from diffusers.utils.torch_utils import randn_tensor +import fastvideo.envs as envs from fastvideo.distributed import get_local_torch_device from fastvideo.fastvideo_args import FastVideoArgs from fastvideo.image_processor import ImageProcessor @@ -475,7 +475,7 @@ class Cosmos25LatentPreparationStage(CosmosLatentPreparationStage): ) video = torch.cat([cond_video, padding], dim=2) - if os.environ.get("FASTVIDEO_COSMOS25_LOG_KNOBS", "0") in ("1", "true", "True"): + if envs.FASTVIDEO_COSMOS25_LOG_KNOBS.get(): logger.info( "[Cosmos2.5 latent_prep] using conditioning input: image=%s video_latent=%s video_path=%s | " "num_video_frames=%s num_cond_frames=%s num_frames=%s", diff --git a/fastvideo/tests/contract/test_env_policy.py b/fastvideo/tests/contract/test_env_policy.py index 571e22228..99f882ed1 100644 --- a/fastvideo/tests/contract/test_env_policy.py +++ b/fastvideo/tests/contract/test_env_policy.py @@ -6,7 +6,9 @@ The test parses every Python file under fastvideo/ (except fastvideo/third_party and the registry fastvideo/envs.py) with ``ast`` and reports code that: - reads the environment directly for a name outside EXTERNAL_ALLOWLIST; -- writes the environment directly (os.environ, os.putenv, monkeypatch.setenv); +- writes the environment directly (os.environ, os.putenv, monkeypatch.setenv), + or through the envs.*_external helpers for a name outside + EXTERNAL_WRITE_ALLOWLIST; - uses the whole environment (os.environ.copy(), dict(os.environ), patch.dict); - uses a registry field without calling one of its methods (``envs.X == "a"``, ``getter = envs.X.get``); @@ -61,8 +63,33 @@ EXTERNAL_ALLOWLIST = { "PATH": "Executable search path.", "HF_TOKEN": "Token variable that huggingface_hub reads.", "HUGGING_FACE_HUB_TOKEN": "Token variable that huggingface_hub reads.", + "GEMINI_API_KEY": "Gemini API key for the judge metrics.", + "GOOGLE_API_KEY": "Google API key, which the Gemini SDK also accepts.", + "CEREBRAS_API_KEY": "Cerebras API key for the streaming prompt provider.", + "GROQ_API_KEY": "Groq API key for the streaming prompt provider.", + "RAY_USAGE_STATS_ENABLED": "Ray usage-statistics switch.", } +# Variables that FastVideo sets for other tools. Code writes them only through +# envs.set_external, envs.setdefault_external, or envs.unset_external, with a +# literal name. +EXTERNAL_WRITE_ALLOWLIST = { + "RANK": "torch.distributed rendezvous for single-process runs and workers.", + "LOCAL_RANK": "torch.distributed rendezvous for single-process runs and workers.", + "WORLD_SIZE": "torch.distributed rendezvous for single-process runs and workers.", + "MASTER_ADDR": "torch.distributed rendezvous for single-process runs.", + "MASTER_PORT": "torch.distributed rendezvous for single-process runs.", + "CUDA_VISIBLE_DEVICES": "Pins a streaming worker process to its GPU.", + "TORCH_NCCL_AVOID_RECORD_STREAMS": "Avoids NCCL record-stream memory growth in workers.", + "NCCL_ASYNC_ERROR_HANDLING": "Unset in workers because the value that Ray sets breaks graph building.", + "TORCH_HOME": "Points torch.hub at the evaluation cache.", + "HF_TOKEN": "Passes the resolved Hugging Face token to huggingface_hub.", + "VIDEO_MAX_PIXELS": "Pixel budget that qwen_vl_utils reads for Qwen2.5-Omni.", + "RAY_USAGE_STATS_ENABLED": "Turns off Ray usage statistics unless the user turned them on.", + "CUTE_DSL_ENABLE_TVM_FFI": "CUTLASS DSL switch that the NVFP4 FlashAttention-4 path needs.", +} +EXTERNAL_WRITE_HELPERS = {"set_external", "setdefault_external", "unset_external"} + # Methods that registry fields expose; ``get`` and ``is_set`` count as reads. REGISTRY_METHODS = {"get", "set", "override", "is_set", "clear"} REGISTRY_READ_METHODS = {"get", "is_set"} @@ -72,111 +99,17 @@ REGISTRY_READ_METHODS = {"get", "is_set"} # its violations are fixed; never add one. Kinds are described in # docs/contributing/env_vars.md. KNOWN_VIOLATIONS: dict[str, int] = { - 'fastvideo/attention/backends/flash_attn.py: read FASTVIDEO_NVFP4_FA4': 1, - 'fastvideo/attention/backends/video_sparse_attn_h3_probe.py: read FASTVIDEO_H3_VSA_PROBE': 1, - 'fastvideo/attention/layer.py: read FASTVIDEO_DISABLE_ATTENTION_COMPILE': 2, - 'fastvideo/attention/selector.py: read ': 1, 'fastvideo/attention/utils/flash_attn_default.py: import-time-read FASTVIDEO_FA4': 1, 'fastvideo/benchmarks/eval_metalfx_rife.py: write FASTVIDEO_MLX_COMPILE': 1, - 'fastvideo/benchmarks/mlx_fastwan_bench.py: read FASTVIDEO_MLX_COMPILE': 1, - 'fastvideo/benchmarks/mlx_fastwan_bench.py: read FASTVIDEO_MLX_FAST_NORM': 1, 'fastvideo/benchmarks/mlx_fastwan_bench.py: write FASTVIDEO_MLX_COMPILE': 1, 'fastvideo/distributed/device_communicators/cpu_communicator.py: read VLLM_DIST_IDENT': 1, 'fastvideo/entrypoints/cli/utils.py: whole-environ': 1, 'fastvideo/entrypoints/openai/api_server.py: write FASTVIDEO_STAGE_LOGGING': 1, - 'fastvideo/entrypoints/streaming/prompt/providers/cerebras.py: read ': 1, - 'fastvideo/entrypoints/streaming/prompt/providers/groq.py: read ': 1, - 'fastvideo/entrypoints/streaming/worker.py: write CUDA_VISIBLE_DEVICES': 1, - 'fastvideo/entrypoints/video_generator.py: read FASTVIDEO_FFMPEG_BIN': 1, - 'fastvideo/entrypoints/video_generator.py: read FASTVIDEO_NVENC_BF': 1, - 'fastvideo/entrypoints/video_generator.py: read FASTVIDEO_NVENC_PRESET': 1, - 'fastvideo/entrypoints/video_generator.py: read FASTVIDEO_NVENC_QP': 1, - 'fastvideo/entrypoints/video_generator.py: read FASTVIDEO_NVENC_RC': 1, - 'fastvideo/entrypoints/video_generator.py: read FASTVIDEO_NVENC_TUNE': 1, - 'fastvideo/entrypoints/video_generator.py: read FASTVIDEO_OUTPUT_PIX_FMT': 1, - 'fastvideo/entrypoints/video_generator.py: read FASTVIDEO_VIDEO_CODEC': 1, - 'fastvideo/entrypoints/video_generator.py: read FASTVIDEO_X264_PRESET': 1, - 'fastvideo/entrypoints/video_generator.py: write CUTE_DSL_ENABLE_TVM_FFI': 1, 'fastvideo/entrypoints/video_generator.py: write FASTVIDEO_NVFP4_FA4': 1, - 'fastvideo/envs.py: prefix CMAKE_BUILD_TYPE': 1, - 'fastvideo/envs.py: prefix CUDA_VISIBLE_DEVICES': 1, - 'fastvideo/envs.py: prefix HCCL_SO_PATH': 1, - 'fastvideo/envs.py: prefix LD_LIBRARY_PATH': 1, - 'fastvideo/envs.py: prefix LOCAL_RANK': 1, - 'fastvideo/envs.py: prefix MAX_JOBS': 1, - 'fastvideo/envs.py: prefix NVCC_THREADS': 1, - 'fastvideo/envs.py: prefix VERBOSE': 1, - 'fastvideo/envs.py: unread CMAKE_BUILD_TYPE': 1, - 'fastvideo/envs.py: unread CUDA_VISIBLE_DEVICES': 1, - 'fastvideo/envs.py: unread FASTVIDEO_ENGINE_ITERATION_TIMEOUT_S': 1, - 'fastvideo/envs.py: unread FASTVIDEO_RINGBUFFER_WARNING_INTERVAL': 1, - 'fastvideo/envs.py: unread FASTVIDEO_SERVER_DEV_MODE': 1, - 'fastvideo/envs.py: unread FASTVIDEO_TARGET_DEVICE': 1, - 'fastvideo/envs.py: unread FASTVIDEO_TEST_DYNAMO_FULLGRAPH_CAPTURE': 1, - 'fastvideo/envs.py: unread FASTVIDEO_TRACE_FUNCTION': 1, - 'fastvideo/envs.py: unread FASTVIDEO_USE_PRECOMPILED': 1, - 'fastvideo/envs.py: unread LD_LIBRARY_PATH': 1, - 'fastvideo/envs.py: unread MAX_JOBS': 1, - 'fastvideo/envs.py: unread NVCC_THREADS': 1, - 'fastvideo/envs.py: unread VERBOSE': 1, - 'fastvideo/eval/__init__.py: write TORCH_HOME': 1, - 'fastvideo/eval/datasets/physics_iq.py: read FASTVIDEO_PHYSICS_IQ_BUCKET_URL': 1, - 'fastvideo/eval/datasets/vbench.py: read VBENCH_FULL_INFO_JSON': 1, - 'fastvideo/eval/metrics/audio/frechet_distance/metric.py: read ': 1, - 'fastvideo/eval/metrics/common/fvd/metric.py: read ': 1, - 'fastvideo/eval/metrics/judge/third_person_separation/metric.py: read ': 1, - 'fastvideo/eval/metrics/vbench/scene/metric.py: write VIDEO_MAX_PIXELS': 1, - 'fastvideo/eval/models.py: read FASTVIDEO_EVAL_CACHE': 1, - 'fastvideo/logger.py: import-time-read FASTVIDEO_CONFIGURE_LOGGING': 1, - 'fastvideo/logger.py: import-time-read FASTVIDEO_LOGGING_CONFIG_PATH': 1, - 'fastvideo/logger.py: import-time-read FASTVIDEO_LOGGING_LEVEL': 1, - 'fastvideo/logger.py: import-time-read FASTVIDEO_LOGGING_PREFIX': 1, - 'fastvideo/mlx_runtime/fastwan.py: read FASTVIDEO_MLX_COMPILE': 1, - 'fastvideo/mlx_runtime/fastwan.py: read FASTVIDEO_MLX_DQ_GEMM': 1, - 'fastvideo/mlx_runtime/fastwan.py: read FASTVIDEO_MLX_FAST_NORM': 1, - 'fastvideo/mlx_runtime/fastwan.py: read FASTVIDEO_MLX_WINDOW': 1, - 'fastvideo/mlx_runtime/fastwan.py: read FASTVIDEO_MLX_WINDOW_SINK': 1, 'fastvideo/mlx_runtime/memory.py: write ': 1, - 'fastvideo/mlx_runtime/wan22.py: read FASTVIDEO_MLX_COMPILE': 1, - 'fastvideo/models/encoders/gemma.py: read LTX2_FASTVIDEO_GEMMA_LOG': 2, - 'fastvideo/models/encoders/reason1.py: read FASTVIDEO_REASON1_WEIGHTS_PATH': 1, - 'fastvideo/models/loader/benchmarks/benchmark_weight_loading.py: write MASTER_ADDR': 1, - 'fastvideo/models/loader/benchmarks/benchmark_weight_loading.py: write MASTER_PORT': 1, - 'fastvideo/models/loader/benchmarks/benchmark_weight_loading.py: write RANK': 1, - 'fastvideo/models/loader/benchmarks/benchmark_weight_loading.py: write WORLD_SIZE': 1, - 'fastvideo/models/loader/fsdp_load.py: read FASTVIDEO_FSDP2_AUTOWRAP': 1, - 'fastvideo/models/loader/fsdp_load.py: read FASTVIDEO_FSDP2_MIN_PARAMS': 1, - 'fastvideo/models/loader/fsdp_load.py: read FASTVIDEO_H3_VSA_PROBE': 1, - 'fastvideo/models/vaes/ltx2vae.py: read ': 1, 'fastvideo/performance/hf_store.py: read HF_REPO_ID': 1, 'fastvideo/performance/hf_store.py: read PERFORMANCE_TRACKING_SYNC_REUSE_TTL_SECONDS': 1, 'fastvideo/performance_dashboard/api.py: read PERFORMANCE_TRACKING_ROOT': 1, - 'fastvideo/pipelines/basic/ltx2/stages/ltx2_audio_decoding.py: read LTX2_DISABLE_AUDIO_AUTOCAST': 1, - 'fastvideo/pipelines/basic/ltx2/stages/ltx2_denoising.py: read FASTVIDEO_NVTX_PROFILE': 1, - 'fastvideo/pipelines/basic/ltx2/stages/ltx2_denoising.py: read LTX2_USE_DISTILLED_SIGMAS': 1, - 'fastvideo/pipelines/basic/magi_human/magi_human_pipeline.py: write HF_TOKEN': 1, - 'fastvideo/pipelines/preprocess/preprocess_kandinsky5_overfit.py: read KANDINSKY5_OVERFIT_DATA_DIR': 1, - 'fastvideo/pipelines/preprocess/preprocess_kandinsky5_overfit.py: read KANDINSKY5_OVERFIT_OUTPUT_DIR': 1, - 'fastvideo/pipelines/preprocess/preprocess_kandinsky5_overfit.py: write MASTER_ADDR': 1, - 'fastvideo/pipelines/preprocess/preprocess_kandinsky5_overfit.py: write MASTER_PORT': 1, - 'fastvideo/pipelines/preprocess/preprocess_ltx2_overfit.py: read LTX2_OVERFIT_CAPTION_JSON': 1, - 'fastvideo/pipelines/preprocess/preprocess_ltx2_overfit.py: read LTX2_OVERFIT_DATA_DIR': 1, - 'fastvideo/pipelines/preprocess/preprocess_ltx2_overfit.py: read LTX2_OVERFIT_MODEL': 1, - 'fastvideo/pipelines/preprocess/preprocess_ltx2_overfit.py: read LTX2_OVERFIT_NUM_COPIES': 1, - 'fastvideo/pipelines/preprocess/preprocess_ltx2_overfit.py: read LTX2_OVERFIT_OUTPUT_DIR': 1, - 'fastvideo/pipelines/preprocess/preprocess_ltx2_overfit.py: read LTX2_OVERFIT_VIDEO_SUBDIR': 1, - 'fastvideo/pipelines/preprocess/preprocess_ltx2_overfit.py: write LOCAL_RANK': 1, - 'fastvideo/pipelines/preprocess/preprocess_ltx2_overfit.py: write MASTER_ADDR': 1, - 'fastvideo/pipelines/preprocess/preprocess_ltx2_overfit.py: write MASTER_PORT': 1, - 'fastvideo/pipelines/preprocess/preprocess_ltx2_overfit.py: write RANK': 1, - 'fastvideo/pipelines/preprocess/preprocess_ltx2_overfit.py: write WORLD_SIZE': 1, - 'fastvideo/pipelines/preprocess/preprocess_minimax_h3_overfit.py: write LOCAL_RANK': 1, - 'fastvideo/pipelines/preprocess/preprocess_minimax_h3_overfit.py: write MASTER_ADDR': 1, - 'fastvideo/pipelines/preprocess/preprocess_minimax_h3_overfit.py: write MASTER_PORT': 1, - 'fastvideo/pipelines/preprocess/preprocess_minimax_h3_overfit.py: write RANK': 1, - 'fastvideo/pipelines/preprocess/preprocess_minimax_h3_overfit.py: write WORLD_SIZE': 1, - 'fastvideo/pipelines/stages/denoising.py: read FASTVIDEO_FLUX2_DISABLE_BF16_REDUCED_PRECISION_REDUCTION': 1, - 'fastvideo/pipelines/stages/latent_preparation.py: read FASTVIDEO_COSMOS25_LOG_KNOBS': 1, 'fastvideo/tests/api/test_attention_selector_resolution.py: write FASTVIDEO_ATTENTION_BACKEND': 17, 'fastvideo/tests/attention/test_flash_attn_no_pad_resolver.py: write FASTVIDEO_FA4': 3, 'fastvideo/tests/attention/test_flash_attn_nvfp4_opt_out.py: write FASTVIDEO_NVFP4_FA4': 4, @@ -345,8 +278,8 @@ KNOWN_VIOLATIONS: dict[str, int] = { 'fastvideo/tests/modal/test_pr_test.py: write BUILDKITE_REPO': 2, 'fastvideo/tests/nightly/test_e2e_dmd_t2v_crush_smol.py: write WANDB_MODE': 1, 'fastvideo/tests/nightly/test_e2e_i2v_overfit_single_sample.py: write WANDB_MODE': 1, - 'fastvideo/tests/nightly/test_e2e_kandinsky5_dmd_t2v_overfit.py: read ': 1, 'fastvideo/tests/nightly/test_e2e_kandinsky5_dmd_t2v_overfit.py: read KANDINSKY5_E2E_NUM_GPUS': 1, + 'fastvideo/tests/nightly/test_e2e_kandinsky5_dmd_t2v_overfit.py: read KANDINSKY5_E2E_WRITE_REFERENCE': 1, 'fastvideo/tests/nightly/test_e2e_kandinsky5_dmd_t2v_overfit.py: whole-environ': 3, 'fastvideo/tests/nightly/test_e2e_kandinsky5_dmd_t2v_overfit.py: write WANDB_MODE': 1, 'fastvideo/tests/nightly/test_e2e_ltx2_overfit_new_stack.py: read FASTVIDEO_NIGHTLY': 1, @@ -420,8 +353,10 @@ KNOWN_VIOLATIONS: dict[str, int] = { 'fastvideo/tests/ssim/conftest.py: write ': 3, 'fastvideo/tests/ssim/inference_similarity_utils.py: read FASTVIDEO_ATTENTION_BACKEND': 1, 'fastvideo/tests/ssim/inference_similarity_utils.py: write FASTVIDEO_ATTENTION_BACKEND': 3, - 'fastvideo/tests/ssim/reference_utils.py: read ': 1, - 'fastvideo/tests/ssim/reference_videos_cli.py: read ': 3, + 'fastvideo/tests/ssim/reference_utils.py: read FASTVIDEO_SSIM_FULL_QUALITY': 1, + 'fastvideo/tests/ssim/reference_videos_cli.py: read ': 1, + 'fastvideo/tests/ssim/reference_videos_cli.py: read FASTVIDEO_SSIM_REFERENCE_HF_REPO': 1, + 'fastvideo/tests/ssim/reference_videos_cli.py: read FASTVIDEO_SSIM_REFERENCE_HF_REPO_TYPE': 1, 'fastvideo/tests/ssim/test_dreamx_world_similarity.py: read DREAMX_WORLD_AR_SSIM_MODEL_PATH': 1, 'fastvideo/tests/ssim/test_dreamx_world_similarity.py: read DREAMX_WORLD_SSIM_MODEL_PATH': 1, 'fastvideo/tests/ssim/test_flux_t2i_similarity.py: read FLUX_T2I_MODEL_DIR': 1, @@ -444,7 +379,7 @@ KNOWN_VIOLATIONS: dict[str, int] = { 'fastvideo/tests/stages/_denoising_fixtures.py: write FASTVIDEO_CFG_GATE_STEP': 2, 'fastvideo/tests/stages/test_kandinsky5_dmd_stage_backend_engages.py: write FASTVIDEO_ATTENTION_BACKEND': 1, 'fastvideo/tests/stages/test_minimax_h3_encoding_offload.py: write FASTVIDEO_ATTENTION_BACKEND': 1, - 'fastvideo/tests/train/methods/grad_norm_regression.py: read ': 1, + 'fastvideo/tests/train/methods/grad_norm_regression.py: read FASTVIDEO_GRADNORM_UPDATE': 1, 'fastvideo/tests/train/methods/test_cosmos_finetune.py: write MASTER_ADDR': 1, 'fastvideo/tests/train/methods/test_cosmos_finetune.py: write MASTER_PORT': 1, 'fastvideo/tests/train/methods/test_ltx2_finetune.py: write MASTER_ADDR': 1, @@ -534,23 +469,11 @@ KNOWN_VIOLATIONS: dict[str, int] = { 'fastvideo/tests/vaes/test_wan_vae.py: write MASTER_PORT': 1, 'fastvideo/tests/worker/test_gpu_worker.py: write FASTVIDEO_NVTX_PROFILE': 2, 'fastvideo/tests/worker/test_gpu_worker.py: write LOCAL_RANK': 1, - 'fastvideo/train/entrypoint/dcp_to_diffusers.py: write ': 1, 'fastvideo/train/entrypoint/train.py: write FASTVIDEO_ATTENTION_BACKEND': 2, - 'fastvideo/training/self_forcing_distillation_pipeline.py: read FASTVIDEO_FSDP2_AUTOWRAP': 1, 'fastvideo/utils.py: read ': 5, - 'fastvideo/utils.py: read FASTVIDEO_WORKER_MULTIPROC_METHOD': 1, 'fastvideo/utils.py: write ': 1, - 'fastvideo/utils.py: write FASTVIDEO_WORKER_MULTIPROC_METHOD': 1, - 'fastvideo/worker/gpu_worker.py: write LOCAL_RANK': 1, - 'fastvideo/worker/gpu_worker.py: write NCCL_ASYNC_ERROR_HANDLING': 1, - 'fastvideo/worker/gpu_worker.py: write RANK': 1, - 'fastvideo/worker/gpu_worker.py: write TORCH_NCCL_AVOID_RECORD_STREAMS': 1, - 'fastvideo/worker/gpu_worker.py: write WORLD_SIZE': 1, 'fastvideo/worker/ray_distributed_executor.py: read ': 2, - 'fastvideo/worker/ray_distributed_executor.py: read RAY_USAGE_STATS_ENABLED': 1, 'fastvideo/worker/ray_distributed_executor.py: whole-environ': 1, - 'fastvideo/worker/ray_distributed_executor.py: write RAY_USAGE_STATS_ENABLED': 1, - 'fastvideo/worker/ray_env.py: import-time-read FASTVIDEO_CONFIG_ROOT': 1, 'fastvideo/worker/ray_env.py: read ': 1, 'fastvideo/worker/ray_utils.py: whole-environ': 1, 'fastvideo/worker/worker_base.py: read ': 1, @@ -576,12 +499,6 @@ def is_allowlisted(name: str) -> bool: return False -def _literal(node: ast.AST | None) -> str: - if isinstance(node, ast.Constant) and isinstance(node.value, str): - return node.value - return "" - - class EnvAccessScanner: """Find environment accesses and registry-field uses in one module. @@ -604,6 +521,13 @@ class EnvAccessScanner: self.getenv_names: set[str] = set() self.setenv_names: set[str] = set() self.envs_module_names: set[str] = set() + # Module-level string constants, so that os.environ.get(NAME_ENV) resolves to the name. + self.constants: dict[str, str] = {} + for statement in tree.body: + if (isinstance(statement, ast.Assign) and len(statement.targets) == 1 + and isinstance(statement.targets[0], ast.Name) and isinstance(statement.value, ast.Constant) + and isinstance(statement.value.value, str)): + self.constants[statement.targets[0].id] = statement.value.value for node in ast.walk(tree): if isinstance(node, ast.Import): for alias in node.names: @@ -624,6 +548,14 @@ class EnvAccessScanner: elif node.module == "fastvideo" and alias.name == "envs": self.envs_module_names.add(local) + def _literal(self, node: ast.AST | None) -> str: + """Return the variable name that a call argument spells, or "".""" + if isinstance(node, ast.Constant) and isinstance(node.value, str): + return node.value + if isinstance(node, ast.Name) and node.id in self.constants: + return self.constants[node.id] + return "" + def _is_os_attr(self, node: ast.AST, attrs: tuple[str, ...]) -> bool: return (isinstance(node, ast.Attribute) and node.attr in attrs and isinstance(node.value, ast.Name) and node.value.id in self.os_names) @@ -670,24 +602,24 @@ class EnvAccessScanner: call = self.parents[parent] assert isinstance(call, ast.Call) if call.func is parent and parent.attr == "get": - self.violations.append(("read", _literal(call.args[0] if call.args else None), line)) + self.violations.append(("read", self._literal(call.args[0] if call.args else None), line)) return if call.func is parent and parent.attr in ("setdefault", "pop"): - self.violations.append(("write", _literal(call.args[0] if call.args else None), line)) + self.violations.append(("write", self._literal(call.args[0] if call.args else None), line)) return if isinstance(parent, ast.Subscript) and parent.value is node: kind = "read" if isinstance(parent.ctx, ast.Load) else "write" - self.violations.append((kind, _literal(parent.slice), line)) + self.violations.append((kind, self._literal(parent.slice), line)) return if (isinstance(parent, ast.Compare) and len(parent.ops) == 1 and isinstance(parent.ops[0], (ast.In, ast.NotIn)) and parent.comparators[0] is node): - self.violations.append(("read", _literal(parent.left), line)) + self.violations.append(("read", self._literal(parent.left), line)) return self.violations.append(("whole-environ", "", line)) def _check_call(self, node: ast.Call) -> None: func = node.func - name = _literal(node.args[0] if node.args else None) + name = self._literal(node.args[0] if node.args else None) if self._is_os_attr(func, ("getenv", "getenvb")) or (isinstance(func, ast.Name) and func.id in self.getenv_names): self.violations.append(("read", name, node.lineno)) @@ -695,6 +627,9 @@ class EnvAccessScanner: or (isinstance(func, ast.Name) and func.id in self.setenv_names) or (isinstance(func, ast.Attribute) and func.attr in ("setenv", "delenv"))): self.violations.append(("write", name, node.lineno)) + elif (isinstance(func, ast.Attribute) and func.attr in EXTERNAL_WRITE_HELPERS + and self._is_envs_module(func.value) and name not in EXTERNAL_WRITE_ALLOWLIST): + self.violations.append(("write", name, node.lineno)) def _check_registry_use(self, node: ast.Attribute, parent: ast.AST | None, in_function: bool) -> None: # Only a call counts: ``getter = envs.X.get`` neither reads the variable @@ -715,11 +650,21 @@ def scanned_files() -> list[Path]: if path != REGISTRY_PATH and not any(excluded in path.parents for excluded in EXCLUDED_DIRS)) +def registry_internal_reads(registry_names: set[str]) -> Counter[str]: + """Count registry reads inside fastvideo/envs.py itself, such as a computed default that reads another field.""" + reads: Counter[str] = Counter() + for node in ast.walk(ast.parse(REGISTRY_PATH.read_text(encoding="utf-8"))): + if (isinstance(node, ast.Attribute) and node.attr in REGISTRY_READ_METHODS and isinstance(node.value, ast.Name) + and node.value.id in registry_names): + reads[node.value.id] += 1 + return reads + + def collect_violations(registry: ModuleType) -> tuple[dict[str, list[int]], Counter[str]]: """Return every violation key with its line numbers, plus registry read counts.""" registry_names = set(registry.environment_variables) found: dict[str, list[int]] = {} - reads: Counter[str] = Counter() + reads: Counter[str] = registry_internal_reads(registry_names) for path in scanned_files(): relative = path.relative_to(REPO_ROOT).as_posix() scanner = EnvAccessScanner(ast.parse(path.read_text(encoding="utf-8"), filename=relative), registry_names) @@ -793,10 +738,21 @@ def _escape(text: str) -> str: def render_env_table(registry: ModuleType) -> str: - """Render the registry as an aligned Markdown table, in declaration order.""" + """Render the registry and the deprecation table as aligned Markdown tables, in declaration order.""" rows = [["Variable", "Type", "Default", "Category", "Description"]] for name, field in registry.environment_variables.items(): - rows.append([f"`{name}`", field.type_name, _render_default(field), field.category, _escape(field.doc)]) + doc = _escape(field.doc) + if field.deprecated_names: + doc += " Deprecated names: " + ", ".join(f"`{old}`" for old in field.deprecated_names) + "." + rows.append([f"`{name}`", field.type_name, _render_default(field), field.category, doc]) + deprecated_rows = [["Deprecated variable", "Reason"]] + deprecated_rows.extend([f"`{name}`", reason] for name, reason in registry.DEPRECATED_VARIABLES.items()) + return (_render_table(rows) + "\n\nVariables that FastVideo no longer reads; setting one logs a warning:\n\n" + + _render_table(deprecated_rows)) + + +def _render_table(rows: list[list[str]]) -> str: + """Render rows as a Markdown table whose pipes line up.""" widths = [max(len(row[column]) for row in rows) for column in range(len(rows[0]))] def render_row(cells: list[str]) -> str: diff --git a/fastvideo/tests/nightly/test_e2e_kandinsky5_dmd_t2v_overfit.py b/fastvideo/tests/nightly/test_e2e_kandinsky5_dmd_t2v_overfit.py index 43971b7fe..bd9f06b18 100644 --- a/fastvideo/tests/nightly/test_e2e_kandinsky5_dmd_t2v_overfit.py +++ b/fastvideo/tests/nightly/test_e2e_kandinsky5_dmd_t2v_overfit.py @@ -84,7 +84,7 @@ NUM_GPUS = os.environ.get("KANDINSKY5_E2E_NUM_GPUS", "1") # with their real videos/captions for the actual recipe: # _clean_previous_artifacts() deletes this root wholesale on every run, # and the preprocess subprocess is pointed here via the -# KANDINSKY5_OVERFIT_DATA_DIR / KANDINSKY5_OVERFIT_OUTPUT_DIR env +# FASTVIDEO_TEST_KANDINSKY5_OVERFIT_DATA_DIR / FASTVIDEO_TEST_KANDINSKY5_OVERFIT_OUTPUT_DIR env # overrides instead of the script's user-facing defaults. E2E_ROOT = Path("data") / "kandinsky5_e2e" RAW_DATA_DIR = E2E_ROOT / "raw" @@ -223,8 +223,8 @@ def _run_preprocessing() -> None: # (data/kandinsky5_overfit{,_preprocessed}) -- the very directories a # user populates for the real recipe, which this test must never touch. env = dict(os.environ) - env["KANDINSKY5_OVERFIT_DATA_DIR"] = str(RAW_DATA_DIR) - env["KANDINSKY5_OVERFIT_OUTPUT_DIR"] = str(PREPROCESSED_DIR) + env["FASTVIDEO_TEST_KANDINSKY5_OVERFIT_DATA_DIR"] = str(RAW_DATA_DIR) + env["FASTVIDEO_TEST_KANDINSKY5_OVERFIT_OUTPUT_DIR"] = str(PREPROCESSED_DIR) cmd = [ sys.executable, "-m", diff --git a/fastvideo/tests/nightly/test_e2e_ltx2_overfit_new_stack.py b/fastvideo/tests/nightly/test_e2e_ltx2_overfit_new_stack.py index 5ffa76e82..476bc55c3 100644 --- a/fastvideo/tests/nightly/test_e2e_ltx2_overfit_new_stack.py +++ b/fastvideo/tests/nightly/test_e2e_ltx2_overfit_new_stack.py @@ -82,9 +82,9 @@ def download_data(): def run_preprocessing(case: dict): env = os.environ.copy() env["CUDA_VISIBLE_DEVICES"] = env.get("CUDA_VISIBLE_DEVICES", "0").split(",")[0] - env["LTX2_OVERFIT_DATA_DIR"] = str(LOCAL_RAW_DATA_DIR) - env["LTX2_OVERFIT_OUTPUT_DIR"] = str(case["prep_dir"]) - env["LTX2_OVERFIT_MODEL"] = case["model"] + env["FASTVIDEO_TEST_LTX2_OVERFIT_DATA_DIR"] = str(LOCAL_RAW_DATA_DIR) + env["FASTVIDEO_TEST_LTX2_OVERFIT_OUTPUT_DIR"] = str(case["prep_dir"]) + env["FASTVIDEO_TEST_LTX2_OVERFIT_MODEL"] = case["model"] cmd = [sys.executable, PREPROCESSING_SCRIPT] print(f"Running preprocessing: {cmd} (model={case['model']})") subprocess.run(cmd, check=True, env=env) diff --git a/fastvideo/train/entrypoint/dcp_to_diffusers.py b/fastvideo/train/entrypoint/dcp_to_diffusers.py index da9d8e0b5..ec07a5412 100644 --- a/fastvideo/train/entrypoint/dcp_to_diffusers.py +++ b/fastvideo/train/entrypoint/dcp_to_diffusers.py @@ -33,6 +33,7 @@ import os import sys from typing import Any +import fastvideo.envs as envs from fastvideo.logger import init_logger logger = init_logger(__name__) @@ -45,14 +46,11 @@ def _ensure_distributed() -> None: For plain ``python`` we fill in the minimum required vars so that ``init_process_group`` succeeds with world_size=1. """ - for key, default in [ - ("RANK", "0"), - ("LOCAL_RANK", "0"), - ("WORLD_SIZE", "1"), - ("MASTER_ADDR", "127.0.0.1"), - ("MASTER_PORT", "29500"), - ]: - os.environ.setdefault(key, default) + envs.setdefault_external("RANK", "0") + envs.setdefault_external("LOCAL_RANK", "0") + envs.setdefault_external("WORLD_SIZE", "1") + envs.setdefault_external("MASTER_ADDR", "127.0.0.1") + envs.setdefault_external("MASTER_PORT", "29500") def _save_role_pretrained( diff --git a/fastvideo/train/models/ltx2/ltx2.py b/fastvideo/train/models/ltx2/ltx2.py index c36fe200b..d73c40a7b 100644 --- a/fastvideo/train/models/ltx2/ltx2.py +++ b/fastvideo/train/models/ltx2/ltx2.py @@ -396,7 +396,7 @@ class LTX2Model(WanModel): f"checkpoint's expected text context dim {expected_dim}. " "The parquet was likely preprocessed with a different " "LTX-2 version (2.0 stores 3840-d, 2.3 stores 4096-d); " - "re-run preprocess_ltx2_overfit.py with LTX2_OVERFIT_MODEL " + "re-run preprocess_ltx2_overfit.py with FASTVIDEO_TEST_LTX2_OVERFIT_MODEL " "set to the same checkpoint as models.student.init_from.") def _update_rope_fps(self, infos: list[dict[str, Any]] | None) -> None: diff --git a/fastvideo/training/self_forcing_distillation_pipeline.py b/fastvideo/training/self_forcing_distillation_pipeline.py index bbc5a44d6..e1db17c70 100644 --- a/fastvideo/training/self_forcing_distillation_pipeline.py +++ b/fastvideo/training/self_forcing_distillation_pipeline.py @@ -1,6 +1,5 @@ # SPDX-License-Identifier: Apache-2.0 import copy -import os import time from collections import deque from typing import Any @@ -47,7 +46,7 @@ class SelfForcingDistillationPipeline(DistillationPipeline): def initialize_training_pipeline(self, training_args: TrainingArgs): """Initialize the self-forcing training pipeline.""" # Check if FSDP2 auto wrap is enabled - not supported for self-forcing distillation - if os.environ.get("FASTVIDEO_FSDP2_AUTOWRAP", "0") == "1": + if envs.FASTVIDEO_FSDP2_AUTOWRAP.get(): raise NotImplementedError("FASTVIDEO_FSDP2_AUTOWRAP is not implemented for self-forcing distillation. " "Please set FASTVIDEO_FSDP2_AUTOWRAP=0 or unset the environment variable.") diff --git a/fastvideo/utils.py b/fastvideo/utils.py index 441ac3fcd..b42c3f606 100644 --- a/fastvideo/utils.py +++ b/fastvideo/utils.py @@ -82,16 +82,16 @@ def find_nccl_library() -> str: def find_hccl_library() -> str: """ - We either use the library file specified by the `HCCL_SO_PATH` + We either use the library file specified by the `FASTVIDEO_HCCL_SO_PATH` environment variable, or we find the library file brought by PyTorch. After importing `torch`, `libhccl.so` can be found by `ctypes` automatically. """ - so_file = envs.HCCL_SO_PATH.get() + so_file = envs.FASTVIDEO_HCCL_SO_PATH.get() # manually load the nccl library if so_file: - logger.info("Found hccl from environment variable HCCL_SO_PATH=%s", so_file) + logger.info("Found hccl from environment variable FASTVIDEO_HCCL_SO_PATH=%s", so_file) else: if torch.version.cann is not None: # codespell:ignore cann so_file = "libhccl.so" @@ -1240,9 +1240,9 @@ def xpu_is_initialized() -> bool: def force_spawn() -> None: - if os.environ.get("FASTVIDEO_WORKER_MULTIPROC_METHOD") == "fork": + if envs.FASTVIDEO_WORKER_MULTIPROC_METHOD.get() == "fork": logger.warning("We must use the `spawn` multiprocessing start method.") - os.environ["FASTVIDEO_WORKER_MULTIPROC_METHOD"] = "spawn" + envs.FASTVIDEO_WORKER_MULTIPROC_METHOD.set("spawn") def get_mp_context() -> BaseContext: diff --git a/fastvideo/worker/gpu_worker.py b/fastvideo/worker/gpu_worker.py index 8a22cdf73..d0712091c 100644 --- a/fastvideo/worker/gpu_worker.py +++ b/fastvideo/worker/gpu_worker.py @@ -1,5 +1,4 @@ # SPDX-License-Identifier: Apache-2.0 -import os from typing import Any, cast import torch @@ -50,9 +49,9 @@ class Worker: # this behavior. # Related issue: # https://discuss.pytorch.org/t/cuda-allocation-lifetime-for-inputs-to-distributed-all-reduce/191573 - os.environ["TORCH_NCCL_AVOID_RECORD_STREAMS"] = "1" + envs.set_external("TORCH_NCCL_AVOID_RECORD_STREAMS", "1") # This env var set by Ray causes exceptions with graph building. - os.environ.pop("NCCL_ASYNC_ERROR_HANDLING", None) + envs.unset_external("NCCL_ASYNC_ERROR_HANDLING") # Set environment variables BEFORE calling get_local_torch_device() # so that each worker uses the correct device @@ -60,9 +59,9 @@ class Worker: # Ray deliberately excludes LOCAL_RANK from the copied driver # environment and exposes all GPUs assigned to the node, so leaving an # inherited or missing value here would bind every Ray actor to cuda:0. - os.environ["LOCAL_RANK"] = str(self.local_rank) - os.environ["RANK"] = str(self.rank) - os.environ["WORLD_SIZE"] = str(self.fastvideo_args.num_gpus) + envs.set_external("LOCAL_RANK", str(self.local_rank)) + envs.set_external("RANK", str(self.rank)) + envs.set_external("WORLD_SIZE", str(self.fastvideo_args.num_gpus)) # Platform-agnostic device initialization self.device = get_local_torch_device() diff --git a/fastvideo/worker/ray_distributed_executor.py b/fastvideo/worker/ray_distributed_executor.py index a48e9e506..e7093bd8b 100644 --- a/fastvideo/worker/ray_distributed_executor.py +++ b/fastvideo/worker/ray_distributed_executor.py @@ -94,6 +94,7 @@ class RayDistributedExecutor(Executor): "NCCL_NVLS_ENABLE", "NCCL_DEBUG", "NCCL_DEBUG_SUBSYS", + "LD_LIBRARY_PATH", } def _init_executor(self) -> None: @@ -103,7 +104,7 @@ class RayDistributedExecutor(Executor): # Disable Ray usage stats collection. ray_usage = os.environ.get("RAY_USAGE_STATS_ENABLED", "0") if ray_usage != "1": - os.environ["RAY_USAGE_STATS_ENABLED"] = "0" + envs.set_external("RAY_USAGE_STATS_ENABLED", "0") self._init_workers_ray(placement_group) diff --git a/fastvideo/worker/ray_env.py b/fastvideo/worker/ray_env.py index b3663afbb..4adcee85e 100644 --- a/fastvideo/worker/ray_env.py +++ b/fastvideo/worker/ray_env.py @@ -8,24 +8,20 @@ from fastvideo.logger import init_logger logger = init_logger(__name__) -CONFIG_HOME = envs.FASTVIDEO_CONFIG_ROOT.get() -# This file contains a list of env vars that should not be copied -# from the driver to the Ray workers. -RAY_NON_CARRY_OVER_ENV_VARS_FILE = os.path.join(CONFIG_HOME, "ray_non_carry_over_env_vars.json") - -try: - if os.path.exists(RAY_NON_CARRY_OVER_ENV_VARS_FILE): - with open(RAY_NON_CARRY_OVER_ENV_VARS_FILE) as f: - RAY_NON_CARRY_OVER_ENV_VARS = set(json.load(f)) - else: - RAY_NON_CARRY_OVER_ENV_VARS = set() -except json.JSONDecodeError: - logger.warning( - "Failed to parse %s. Using an empty set for non-carry-over env vars.", - RAY_NON_CARRY_OVER_ENV_VARS_FILE, - ) - RAY_NON_CARRY_OVER_ENV_VARS = set() +def _load_non_carry_over_env_vars(non_carry_over_file: str) -> set[str]: + """Read the env vars that must not be copied from the driver to the Ray workers.""" + try: + if os.path.exists(non_carry_over_file): + with open(non_carry_over_file) as f: + return set(json.load(f)) + return set() + except json.JSONDecodeError: + logger.warning( + "Failed to parse %s. Using an empty set for non-carry-over env vars.", + non_carry_over_file, + ) + return set() def get_env_vars_to_copy( @@ -52,18 +48,26 @@ def get_env_vars_to_copy( """ exclude_vars = exclude_vars or set() additional_vars = additional_vars or set() + # This file contains a list of env vars that should not be copied + # from the driver to the Ray workers. + non_carry_over_file = os.path.join(envs.FASTVIDEO_CONFIG_ROOT.get(), "ray_non_carry_over_env_vars.json") + non_carry_over_vars = _load_non_carry_over_env_vars(non_carry_over_file) + registered_vars = { + name + for field in envs.environment_variables.values() + for name in (field.name, *field.deprecated_names) + } env_vars_to_copy = { v - for v in set(envs.environment_variables).union(additional_vars) - if v not in exclude_vars and v not in RAY_NON_CARRY_OVER_ENV_VARS + for v in registered_vars.union(additional_vars) if v not in exclude_vars and v not in non_carry_over_vars } to_destination = " to " + destination if destination is not None else "" logger.info( "RAY_NON_CARRY_OVER_ENV_VARS from config: %s", - RAY_NON_CARRY_OVER_ENV_VARS, + non_carry_over_vars, ) logger.info( "Copying the following environment variables%s: %s", @@ -72,7 +76,7 @@ def get_env_vars_to_copy( ) logger.info( "If certain env vars should NOT be copied, add them to %s file", - RAY_NON_CARRY_OVER_ENV_VARS_FILE, + non_carry_over_file, ) return env_vars_to_copy diff --git a/tests/local_tests/ltx2/README.md b/tests/local_tests/ltx2/README.md index 33f120be5..4c5501bc9 100644 --- a/tests/local_tests/ltx2/README.md +++ b/tests/local_tests/ltx2/README.md @@ -16,7 +16,7 @@ skip or fail. Skipped in CI; run locally on a single GPU. | Official commit/version | `` | | HF weights | `Lightricks/LTX-2`, `FastVideo/LTX2-base`, `FastVideo/LTX2-Distilled-Diffusers` | | HF revision | `` | -| Local weights dir | `` (env: `LTX2_DIFFUSERS_PATH`, `LTX2_FASTVIDEO_GEMMA_LOG`, etc.) | +| Local weights dir | `` (env: `LTX2_DIFFUSERS_PATH`, `FASTVIDEO_LTX2_GEMMA_LOG`, etc.) | | Source layout | `` | | Needs conversion | `` | diff --git a/tests/local_tests/ltx2/test_ltx2_gemma_parity.py b/tests/local_tests/ltx2/test_ltx2_gemma_parity.py index fd001c77b..acb62e331 100644 --- a/tests/local_tests/ltx2/test_ltx2_gemma_parity.py +++ b/tests/local_tests/ltx2/test_ltx2_gemma_parity.py @@ -21,7 +21,7 @@ if ltx_core_path.exists() and str(ltx_core_path) not in sys.path: def _init_log_paths() -> tuple[Path, Path]: base_dir = Path(os.getenv("LTX2_DEBUG_DIR", "ltx2_debug")) - fastvideo_log = Path(os.getenv("LTX2_FASTVIDEO_GEMMA_LOG", base_dir / "fastvideo_gemma.log")) + fastvideo_log = Path(os.getenv("FASTVIDEO_LTX2_GEMMA_LOG", base_dir / "fastvideo_gemma.log")) reference_log = Path(os.getenv("LTX2_REFERENCE_GEMMA_LOG", base_dir / "reference_gemma.log")) fastvideo_log.parent.mkdir(parents=True, exist_ok=True) reference_log.parent.mkdir(parents=True, exist_ok=True)