hao-ai-lab-FastVideo/docs/cookbook/mmaudio.md

hide

hide
toc

MMAudio recipes

← All model families

MMA

Maintained family · Inference

MMAudio inference recipes

MMAudio adds synchronized audio to video or generates audio from text. The checkpoint must be in Diffusers layout; the recipe loads the converted FastVideo repo through the `MMAUDIO_MODEL_PATH` environment variable the example reads.

Inference live Distillation planned Fine-tuning planned Training planned Evaluation planned Optimization planned Deployment planned

Builder Setup Troubleshooting Evidence

Pick a recipe and runtime

Start with the result you want, then choose one of the runtimes FastVideo actually maintains for it.

<div class="cookbook-builder__layout">
  <div class="cookbook-controls">
    <div class="cookbook-selection-row">
      <div class="cookbook-selection-row__label">
        <strong>Recipe</strong>
        <span>Task and checkpoint</span>
      </div>
      <div class="cookbook-option-grid cookbook-option-grid--models" data-cookbook-model-options role="group" aria-label="Recipe">
        <button type="button" disabled>Loading MMAudio recipes...</button>
      </div>
    </div>

    <div class="cookbook-selection-row">
      <div class="cookbook-selection-row__label">
        <strong>Runtime</strong>
        <span>Maintained paths only</span>
      </div>
      <div class="cookbook-option-grid cookbook-option-grid--hardware" data-cookbook-hardware-options role="group" aria-label="Runtime">
        <button type="button" disabled>Loading runtimes...</button>
      </div>
    </div>

    <p class="cookbook-selection-description" data-cookbook-description>Loading recipe details...</p>
    <p class="cookbook-hardware-note">Exact device and memory details appear only when a recorded run supports them.</p>

    <div class="cookbook-hardware-state" data-cookbook-hardware-state role="status" aria-live="polite">
      Reading recipe evidence...
    </div>
  </div>

  <article class="cookbook-result">
    <div class="cookbook-result__header">
      <h3 data-cookbook-label>Loading...</h3>
      <div class="cookbook-result__badges">
        <span class="cookbook-badge">Maintained</span>
        <span class="cookbook-badge" data-cookbook-evidence>Source-backed</span>
        <span class="cookbook-badge cookbook-badge--neutral" data-cookbook-hardware-badge>Source config</span>
      </div>
    </div>

    <dl class="cookbook-result__facts">
      <div><dt>Model</dt><dd data-cookbook-model>Loading...</dd></div>
      <div><dt>Workload</dt><dd data-cookbook-task>Loading...</dd></div>
      <div><dt>Source configuration</dt><dd data-cookbook-gpus>Loading...</dd></div>
      <div><dt>Expected output</dt><dd data-cookbook-artifact>Loading...</dd></div>
    </dl>

    <div class="cookbook-command">
      <div class="cookbook-command__bar">
        <span>Terminal</span>
      </div>
      <pre><code class="language-bash" data-cookbook-command>Loading...</code></pre>
    </div>

    <div class="cookbook-result__footer">
      <a data-cookbook-source href="../../inference/examples/basic/">Open example source</a>
      <a data-cookbook-model-link href="https://huggingface.co/FastVideo">View model card</a>
    </div>
    <p class="cookbook-picker__status" role="status" aria-live="polite" data-cookbook-status></p>
  </article>
</div>

<noscript>
  <div class="cookbook-noscript">
    JavaScript is needed for the guided selector. You can still browse the
    <a href="../../inference/examples/examples_inference_index/">maintained inference examples</a>.
  </div>
</noscript>

Setup

The generated commands expect a local clone:

git clone https://github.com/hao-ai-lab/FastVideo.git
cd FastVideo

Use Configuration for supported Python and CLI settings, Optimizations for attention and memory tradeoffs, and the support matrix for the supported model and optimization surface.

Troubleshooting

• If the example reports a missing local converted_weights/mmaudio/large_44k_v2 path, the MMAUDIO_MODEL_PATH env var from the cookbook command was not applied; export it in the same shell.
• Alternatively convert upstream weights yourself with scripts/checkpoint_conversion/convert_mmaudio_to_diffusers.py and point the env var at the result.

Evidence status

All recipes on this page are Source-backed: their commands, model IDs, and flags were validated against the checked-in FastVideo sources listed above (static validation). No runtime GPU validation is recorded for these recipes, so GPU model fit, memory use, throughput, and runtime duration are Unknown and deliberately not claimed. Runtime buttons show only the GPU counts configured in checked-in sources.