Davids048andClaude Fable 5.1
a5aa64ab6b
VideoGenerator.from_pretrained(model_path, config) takes the same nested
settings as from_config; the 24 flat keywords, from_pretrained_kwargs_to_config,
FROM_PRETRAINED_KWARGS, and every flat_field in the schema are deleted.
nvfp4_fa4 is the typed field engine.attention.nvfp4_fa4, applied by a
resolution step. 65 call sites and the docs use the nested form; the three
kwargs golden cases are config cases with byte-identical results.
Every value is decided during resolution; no runtime code calls
with_override. The device offload policy (unified memory, MPS, layerwise
conflicts, lazy module load) runs as resolution steps in the main process
(fastvideo/api/device_policy.py), and the worker runs on the config it
receives. The LTX-2 refine defaults from model_index.json and the MiniMax-H3
schedule from fastvideo_inference.json are resolution steps
(fastvideo/api/checkpoint_defaults.py) that read local checkpoints or download
only those files; the H3 pipeline validates the schedule against the loaded
schedulers but no longer writes it. The preprocessing entry point passes the
downloaded local path as run state instead of a config override. Teacher and
critic models load with override_transformer_cls_name as a load argument.
The test isolation turns the device policy off, as it blocks downloads, so
that the goldens do not depend on the machine.
Against the 3f6893a0 goldens, every field still matches except boundary_ratio
and ltx2_vae_tiling (DESIGN section 9) and the fields that no longer exist.
The launcher and trainer YAML verifications report 0 unexplained differences.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
92 lines
3.6 KiB
Python
92 lines
3.6 KiB
Python
# SPDX-License-Identifier: Apache-2.0
|
|
"""Stable Audio Open 1.0 — inpainting / outpainting (loop extension) example.
|
|
|
|
User story (loop extension — the killer app):
|
|
"I have a 6-second drum loop my client likes. They want it as
|
|
background bed for a 30-second ad. I need it to loop seamlessly,
|
|
but a hard cut every 6s sounds bad. Let me extend it to 30s,
|
|
keeping the first 6s exactly as-is and letting the model continue
|
|
the groove for the remaining 24s."
|
|
|
|
User story (audio repair):
|
|
"There's a microphone bump at 0:14 in this 30-second field
|
|
recording — really obvious in headphones. Mask out 0:13 to 0:15
|
|
and let the model regenerate plausible ambience that blends in.
|
|
Everything else stays exactly as I recorded it."
|
|
|
|
User story (transition smoothing):
|
|
"I have two 10-second clips I want to crossfade. Mask out a 1s
|
|
overlap region in the middle and let the model invent a coherent
|
|
transition between the two."
|
|
|
|
How it works (RePaint-style blending):
|
|
Stable Audio Open 1.0 wasn't trained as an inpainting model
|
|
(`model_type=diffusion_cond`, not `diffusion_cond_inpaint`), so we
|
|
can't use the upstream's mask-conditioned approach directly. We
|
|
use the RePaint trick instead, which works on any v-prediction
|
|
diffusion model:
|
|
|
|
1. Encode the reference clip into latent space.
|
|
2. At every denoising step `i`, replace the kept region of the
|
|
in-flight latent (where mask == 1) with the reference
|
|
re-noised to the next timestep's sigma. Only the unkept
|
|
region (mask == 0) is freely denoised.
|
|
3. After the loop, the kept region is exactly the reference;
|
|
the unkept region is freshly generated content.
|
|
|
|
This is approximate compared to a properly trained inpainting
|
|
checkpoint — the seam between kept/unkept can have slight EQ
|
|
discontinuity — but it works on the existing public model.
|
|
|
|
Tunable: the mask is a 1-D tensor in {0, 1} at the model's sample
|
|
rate. Conventions:
|
|
1.0 = keep this sample from the reference
|
|
0.0 = regenerate this sample
|
|
|
|
Prerequisites: same as `basic_stable_audio.py`.
|
|
"""
|
|
import os
|
|
|
|
from fastvideo import VideoGenerator
|
|
|
|
PROMPT = "Steady lo-fi hip hop drum loop with vinyl crackle."
|
|
# Required: path to the reference audio file (wav, mp3, mp4, m4a, flac,
|
|
# ...) you want to extend or repair. The pipeline raises if a mask is
|
|
# passed without a reference, so this must be a real path.
|
|
REFERENCE_AUDIO_PATH = "path/to/your/loop.wav"
|
|
KEEP_SECONDS = 6.0 # first KEEP_SECONDS preserved exactly
|
|
TOTAL_SECONDS = 12.0 # extend the loop to this duration
|
|
|
|
|
|
def main() -> None:
|
|
if not os.path.isfile(REFERENCE_AUDIO_PATH):
|
|
raise FileNotFoundError(f"REFERENCE_AUDIO_PATH={REFERENCE_AUDIO_PATH!r} does not exist. "
|
|
"Edit this script to point at a real audio file (wav/mp3/mp4/"
|
|
"m4a/flac) before running.")
|
|
generator = VideoGenerator.from_pretrained(
|
|
"FastVideo/stable-audio-open-1.0-Diffusers",
|
|
{
|
|
"engine": {
|
|
"num_gpus": 1,
|
|
},
|
|
},
|
|
)
|
|
generator.generate({
|
|
"prompt": PROMPT,
|
|
"output": {
|
|
"output_path": "outputs_audio/stable_audio_inpaint/output_inpaint.wav",
|
|
"save_video": True,
|
|
},
|
|
"extensions": {
|
|
"audio_end_in_s": TOTAL_SECONDS,
|
|
"inpaint_audio": REFERENCE_AUDIO_PATH,
|
|
# Tuple form: keep first KEEP_SECONDS, regenerate the rest.
|
|
"inpaint_mask": (KEEP_SECONDS, TOTAL_SECONDS),
|
|
},
|
|
})
|
|
generator.shutdown()
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|