Improve audio prompt sequencer
This commit is contained in:
Fillip
@@ -394,19 +394,19 @@ Cut to profile as the subject walks toward the doorway.
|
||||
|
||||
Connect `scheduled` to the first sampler for strict latent-time control. Connect `semantic` to a low-denoise upscale pass for normal single-prompt sampling. To enforce the strict schedule again after spatial VAE re-encoding, connect the timeline and resized AV latent to `FL_MiniMaxH3ApplyTimeline`. The Apply node requires the same video and audio duration but supports different spatial dimensions.
|
||||
|
||||
For audio-reactive timing, choose or upload audio directly on `FL_Audio_Beat_Prompt_Schedule`, then connect its `prompt_schedule` output to the H3 timeline node. The scheduler's `beat_positions`, `drum_times`, and `audio` outputs can directly drive the existing FL prompt-envelope, ADSR-envelope, and audio nodes. A connected `beat_positions` input remains an optional timing override for existing workflows:
|
||||
For audio-reactive timing, choose or upload audio directly on `FL_Audio_Beat_Prompt_Schedule`, then connect its `prompt_schedule` output to the H3 timeline node. The scheduler also outputs the selected audio crop, total frame count, and detected BPM. Use the existing FL BPM Analyzer and Drum Detector when another node needs beat-position or drum-event JSON. A connected `beat_positions` input remains an optional timing override for existing workflows:
|
||||
|
||||
```text
|
||||
[0 - 48 | fade_in=6 | fade_out=6]
|
||||
The subject slowly turns toward camera.
|
||||
|
||||
[48 - 96 | fade_in=6 | fade_out=6]
|
||||
[48 - 96 | fade_out=6 | crossfade=12]
|
||||
The camera pushes forward on the beat.
|
||||
```
|
||||
|
||||
Frame ranges are zero-based and range ends are exclusive. Detected beats appear in their own marker lane and can be used as the editor's snap target without changing the stored frame timing. A connected prompt schedule overrides the manual timeline. Exact repeated prompts share one conditioning mask, so reuse prompts for recurring sections such as choruses.
|
||||
Frame ranges are zero-based and range ends are exclusive. `fade_in` and `fade_out` blend a scheduled prompt with the global prompt. `crossfade` belongs to the incoming prompt and blends it directly with a touching previous prompt; the two scheduled weights retain full conditioning coverage throughout the transition. Detected beats appear in their own marker lane and can be used as the editor's snap target without changing the stored frame timing. A connected prompt schedule overrides the manual timeline. Exact repeated prompts share one conditioning mask, so reuse prompts for recurring sections such as choruses.
|
||||
|
||||
The prompt schedule node loads its waveform as soon as an audio file is selected; queueing is not required. Use the source overview handles to trim, Play/Stop for crop-aligned transport, and the main frame ruler to drag or resize prompt clips. Beat Grid, Detected Beat, Onset, Frame, and Off are distinct snap modes. The live Beat offset slider moves the regular and detected beat markers, updates snapping immediately, and leaves the waveform, audio, onsets, drums, and prompt clips fixed. FPS and length remain normal node widgets, and length is always a frame count. Analysis is cached by source hash, crop, FPS, and detector settings; offset changes reuse the same detected timing.
|
||||
The prompt schedule node loads its waveform as soon as an audio file is selected; queueing is not required. Use the source overview handles to trim, Play for continuous crop-aligned looping, Pause/Stop for transport control, and the main frame ruler to drag or resize prompt clips. Drag the shared center boundary between touching prompts to expand one while shortening the other; the pair stays contiguous and the boundary snaps to the beat grid. Right-click a beat-aligned position on the main timeline to set the audio In or Out point; prompt clips are cropped, rebased, and re-snapped with the audio so they stay aligned to the same musical moments. Prompt boundaries also re-snap automatically after crop, FPS, length, grid-density, beat-offset, cached-analysis, or execution timing changes while preserving touching cuts and valid clip lengths. The crop start and end are always valid snap targets, even when the nearest detected beat falls inside the crop. Double-click a shared prompt boundary to toggle a one-beat crossfade, drag its upper handles to resize the transition, or enter an exact frame count in the inspector. The Beat offset slider previews grid movement live and re-snaps the prompt schedule when the adjustment is committed; the waveform, detected reference markers, and audio remain fixed. FPS and length remain normal node widgets, and length is always a frame count. Analysis is cached by source hash, crop, FPS, and detector settings; offset changes reuse the same detected timing.
|
||||
|
||||
`Separate stems` is an explicit action and never runs automatically. It separates the full source once with Hybrid Demucs, caches bass/drums/other/vocals locally, switches analysis to the drums stem, and keeps the node's audio output on the original master crop. Jobs report progress, can be cancelled between chunks, and reuse valid cached stems.
|
||||
|
||||
|
||||
@@ -104,20 +104,25 @@ def _load_beat_data(beat_positions):
|
||||
|
||||
|
||||
def _parse_options(options, line, default_fade_in, default_fade_out):
|
||||
values = {"fade_in": default_fade_in, "fade_out": default_fade_out}
|
||||
values = {
|
||||
"fade_in": default_fade_in,
|
||||
"fade_out": default_fade_out,
|
||||
"crossfade": 0.0,
|
||||
}
|
||||
if not options:
|
||||
return values
|
||||
|
||||
for option in options.split("|"):
|
||||
if "=" not in option:
|
||||
raise ValueError(
|
||||
f"Beat prompt schedule line {line}: options must use fade_in=value or fade_out=value."
|
||||
f"Beat prompt schedule line {line}: options must use "
|
||||
"fade_in=value, fade_out=value, or crossfade=value."
|
||||
)
|
||||
name, value = (part.strip() for part in option.split("=", 1))
|
||||
if name not in values:
|
||||
raise ValueError(
|
||||
f"Beat prompt schedule line {line}: unknown option '{name}'. "
|
||||
"Use fade_in or fade_out."
|
||||
"Use fade_in, fade_out, or crossfade."
|
||||
)
|
||||
values[name] = _number(value, name, line)
|
||||
return values
|
||||
@@ -161,6 +166,7 @@ def _parse_schedule(text, default_fade_in, default_fade_out, time_unit="beats"):
|
||||
"end": end,
|
||||
"fade_in": options["fade_in"],
|
||||
"fade_out": options["fade_out"],
|
||||
"crossfade": options["crossfade"],
|
||||
}
|
||||
for name, value in frame_values.items():
|
||||
if abs(value - round(value)) > _EPS:
|
||||
@@ -210,6 +216,27 @@ def _parse_schedule(text, default_fade_in, default_fade_out, time_unit="beats"):
|
||||
raise ValueError(
|
||||
f"Beat prompt schedule line {section['line']}: section overlaps the previous section."
|
||||
)
|
||||
if section["crossfade"] > _EPS:
|
||||
if abs(section["start_position"] - previous["end_position"]) > _EPS:
|
||||
raise ValueError(
|
||||
f"Beat prompt schedule line {section['line']}: crossfade requires "
|
||||
"a touching previous section."
|
||||
)
|
||||
shortest = min(
|
||||
previous["end_position"] - previous["start_position"],
|
||||
section["end_position"] - section["start_position"],
|
||||
)
|
||||
if section["crossfade"] > shortest + _EPS:
|
||||
raise ValueError(
|
||||
f"Beat prompt schedule line {section['line']}: crossfade exceeds "
|
||||
"the shorter adjacent section."
|
||||
)
|
||||
previous["fade_out"] = 0.0
|
||||
section["fade_in"] = 0.0
|
||||
elif section["crossfade"] > _EPS:
|
||||
raise ValueError(
|
||||
f"Beat prompt schedule line {section['line']}: the first section cannot crossfade."
|
||||
)
|
||||
previous = section
|
||||
return sections
|
||||
|
||||
@@ -284,11 +311,9 @@ def _resolve_schedule(
|
||||
f"Beat prompt schedule line {section['line']}: the selected range resolves "
|
||||
"to an empty time range."
|
||||
)
|
||||
if end > limit + _EPS:
|
||||
raise ValueError(
|
||||
f"Beat prompt schedule line {section['line']}: section ends at {end:g}s, "
|
||||
f"beyond the sequence duration {limit:g}s."
|
||||
)
|
||||
if start >= limit - _EPS:
|
||||
continue
|
||||
end = min(end, limit)
|
||||
fade_in_end = _position_to_seconds(
|
||||
section["start_position"] + section["fade_in"],
|
||||
time_unit,
|
||||
@@ -305,14 +330,50 @@ def _resolve_schedule(
|
||||
fps,
|
||||
section["line"],
|
||||
)
|
||||
fade_in_end = min(end, fade_in_end)
|
||||
fade_out_start = min(end, max(start, fade_out_start))
|
||||
if section["crossfade"] > _EPS:
|
||||
if time_unit == "frames":
|
||||
crossfade_before = math.floor(section["crossfade"] / 2)
|
||||
crossfade_after = section["crossfade"] - crossfade_before
|
||||
else:
|
||||
crossfade_before = section["crossfade"] * 0.5
|
||||
crossfade_after = crossfade_before
|
||||
crossfade_start = _position_to_seconds(
|
||||
section["start_position"] - crossfade_before,
|
||||
time_unit,
|
||||
beat_times,
|
||||
duration,
|
||||
fps,
|
||||
section["line"],
|
||||
)
|
||||
crossfade_end = _position_to_seconds(
|
||||
section["start_position"] + crossfade_after,
|
||||
time_unit,
|
||||
beat_times,
|
||||
duration,
|
||||
fps,
|
||||
section["line"],
|
||||
)
|
||||
crossfade_start = max(0.0, crossfade_start)
|
||||
crossfade_end = min(limit, end, crossfade_end)
|
||||
else:
|
||||
crossfade_start = start
|
||||
crossfade_end = start
|
||||
resolved.append({
|
||||
**section,
|
||||
"start": start,
|
||||
"end": end,
|
||||
"fade_in_end": fade_in_end,
|
||||
"fade_out_start": fade_out_start,
|
||||
"crossfade_start": crossfade_start,
|
||||
"crossfade_end": crossfade_end,
|
||||
"curve": curve,
|
||||
})
|
||||
for previous, section in zip(resolved, resolved[1:]):
|
||||
if section["crossfade_end"] > section["crossfade_start"] + _EPS:
|
||||
previous["fade_out_start"] = previous["end"]
|
||||
section["fade_in_end"] = section["start"]
|
||||
return resolved
|
||||
|
||||
|
||||
@@ -323,39 +384,29 @@ def _frame_sections(sections, fps, total_frames):
|
||||
end_frame = min(total_frames, max(start_frame + 1, round(section["end"] * fps)))
|
||||
fade_in_end = min(end_frame, max(start_frame, round(section["fade_in_end"] * fps)))
|
||||
fade_out_start = min(end_frame, max(start_frame, round(section["fade_out_start"] * fps)))
|
||||
crossfade_start = min(
|
||||
total_frames,
|
||||
max(0, round(section["crossfade_start"] * fps)),
|
||||
)
|
||||
crossfade_end = min(
|
||||
total_frames,
|
||||
max(crossfade_start, round(section["crossfade_end"] * fps)),
|
||||
)
|
||||
frame_sections.append({
|
||||
"line": section["line"],
|
||||
"start_frame": start_frame,
|
||||
"end_frame": end_frame,
|
||||
"fade_in_frames": fade_in_end - start_frame,
|
||||
"fade_out_frames": end_frame - fade_out_start,
|
||||
"crossfade_start_frame": crossfade_start,
|
||||
"crossfade_end_frame": crossfade_end,
|
||||
"crossfade_frames": crossfade_end - crossfade_start,
|
||||
"prompt": section["prompt"],
|
||||
"curve": section["curve"],
|
||||
})
|
||||
return frame_sections
|
||||
|
||||
|
||||
def _format_time(seconds):
|
||||
minutes = int(seconds // 60)
|
||||
remainder = seconds - minutes * 60
|
||||
return f"{minutes:02d}:{remainder:06.3f}"
|
||||
|
||||
|
||||
def _preview(sections, time_unit="beats", fps=24.0):
|
||||
lines = []
|
||||
for section in sections:
|
||||
start_frame = round(section["start"] * fps)
|
||||
end_frame = round(section["end"] * fps)
|
||||
lines.append(
|
||||
f"[{time_unit} {section['start_position']:g} - {section['end_position']:g} | "
|
||||
f"{_format_time(section['start'])} - {_format_time(section['end'])} | "
|
||||
f"frames {start_frame} - {end_frame} @ {fps:g} fps | "
|
||||
f"fade_in={section['fade_in']:g} | fade_out={section['fade_out']:g}]\n"
|
||||
f"{section['prompt']}"
|
||||
)
|
||||
return "\n\n".join(lines)
|
||||
|
||||
|
||||
class FL_Audio_Beat_Prompt_Schedule(io.ComfyNode):
|
||||
@classmethod
|
||||
def define_schema(cls):
|
||||
@@ -518,14 +569,6 @@ class FL_Audio_Beat_Prompt_Schedule(io.ComfyNode):
|
||||
display_name="prompt_schedule",
|
||||
tooltip="Resolved second-based prompt schedule for compatible FL diffusion nodes.",
|
||||
),
|
||||
io.String.Output(
|
||||
display_name="preview",
|
||||
tooltip="Readable preview showing each source range and its resolved time and frame range.",
|
||||
),
|
||||
io.Float.Output(
|
||||
display_name="duration_seconds",
|
||||
tooltip="Effective schedule duration in seconds.",
|
||||
),
|
||||
io.Int.Output(
|
||||
display_name="total_frames",
|
||||
tooltip="Effective schedule duration converted to frames at the selected FPS.",
|
||||
@@ -534,13 +577,9 @@ class FL_Audio_Beat_Prompt_Schedule(io.ComfyNode):
|
||||
display_name="audio",
|
||||
tooltip="The selected, frame-aligned audio crop for downstream FL audio nodes.",
|
||||
),
|
||||
io.String.Output(
|
||||
display_name="beat_positions",
|
||||
tooltip="Effective beat analysis JSON for FL audio-reactive nodes.",
|
||||
),
|
||||
io.String.Output(
|
||||
display_name="drum_times",
|
||||
tooltip="Detected kick, snare, and hi-hat timestamps for FL audio-reactive nodes.",
|
||||
io.Float.Output(
|
||||
display_name="BPM",
|
||||
tooltip="Detected musical tempo after applying the Half-time option.",
|
||||
),
|
||||
],
|
||||
)
|
||||
@@ -638,7 +677,7 @@ class FL_Audio_Beat_Prompt_Schedule(io.ComfyNode):
|
||||
section.update(frame_section)
|
||||
schedule = {
|
||||
"type": "fl_prompt_schedule",
|
||||
"version": 1,
|
||||
"version": 2,
|
||||
"duration": duration,
|
||||
"audio_duration": audio_duration,
|
||||
"source_unit": time_unit,
|
||||
@@ -702,20 +741,11 @@ class FL_Audio_Beat_Prompt_Schedule(io.ComfyNode):
|
||||
)
|
||||
if waveform is not None:
|
||||
ui_payload["waveform_preview"] = waveform
|
||||
drum_times = ui_payload["drum_times"] or {
|
||||
"kick_times": [],
|
||||
"snare_times": [],
|
||||
"hihat_times": [],
|
||||
"duration": duration,
|
||||
}
|
||||
return io.NodeOutput(
|
||||
schedule,
|
||||
_preview(sections, time_unit, fps),
|
||||
duration,
|
||||
total_frames,
|
||||
cropped_audio,
|
||||
beat_positions,
|
||||
json.dumps(drum_times, separators=(",", ":")),
|
||||
float(beat_data["bpm"]),
|
||||
ui={"fl_prompt_sequencer": [ui_payload]},
|
||||
)
|
||||
|
||||
|
||||
@@ -126,10 +126,15 @@ def _resolve_sections(sections, duration, duration_policy, transition_mode, tran
|
||||
end = section["end"] * scale
|
||||
fade_in_end = section.get("fade_in_end")
|
||||
fade_out_start = section.get("fade_out_start")
|
||||
crossfade_start = section.get("crossfade_start")
|
||||
crossfade_end = section.get("crossfade_end")
|
||||
if fade_in_end is not None:
|
||||
fade_in_end *= scale
|
||||
if fade_out_start is not None:
|
||||
fade_out_start *= scale
|
||||
if crossfade_start is not None:
|
||||
crossfade_start *= scale
|
||||
crossfade_end *= scale
|
||||
if duration_policy == "clamp":
|
||||
start = min(start, duration)
|
||||
end = min(end, duration)
|
||||
@@ -137,12 +142,18 @@ def _resolve_sections(sections, duration, duration_policy, transition_mode, tran
|
||||
fade_in_end = min(fade_in_end, duration)
|
||||
if fade_out_start is not None:
|
||||
fade_out_start = min(fade_out_start, duration)
|
||||
if crossfade_start is not None:
|
||||
crossfade_start = min(crossfade_start, duration)
|
||||
crossfade_end = min(crossfade_end, duration)
|
||||
if end <= start + _EPS:
|
||||
continue
|
||||
resolved_section = {**section, "start": start, "end": end}
|
||||
if fade_in_end is not None:
|
||||
resolved_section["fade_in_end"] = min(max(fade_in_end, start), end)
|
||||
resolved_section["fade_out_start"] = min(max(fade_out_start, start), end)
|
||||
if crossfade_start is not None:
|
||||
resolved_section["crossfade_start"] = max(0.0, crossfade_start)
|
||||
resolved_section["crossfade_end"] = min(end, max(crossfade_start, crossfade_end))
|
||||
resolved.append(resolved_section)
|
||||
|
||||
if transition_mode != "hard" and transition_frames > 0:
|
||||
@@ -160,8 +171,9 @@ def _resolve_sections(sections, duration, duration_policy, transition_mode, tran
|
||||
def _schedule_sections(schedule):
|
||||
if not isinstance(schedule, dict) or schedule.get("type") != "fl_prompt_schedule":
|
||||
raise TypeError("FL MiniMax H3 Prompt Timeline received an invalid prompt schedule.")
|
||||
if schedule.get("version") != 1:
|
||||
raise ValueError("FL MiniMax H3 Prompt Timeline supports FL prompt schedule version 1.")
|
||||
version = schedule.get("version")
|
||||
if version not in {1, 2}:
|
||||
raise ValueError("FL MiniMax H3 Prompt Timeline supports FL prompt schedule versions 1 and 2.")
|
||||
|
||||
values = schedule.get("sections")
|
||||
if not isinstance(values, list):
|
||||
@@ -176,13 +188,22 @@ def _schedule_sections(schedule):
|
||||
end = float(value["end"])
|
||||
fade_in_end = float(value["fade_in_end"])
|
||||
fade_out_start = float(value["fade_out_start"])
|
||||
crossfade_start = float(value.get("crossfade_start", start))
|
||||
crossfade_end = float(value.get("crossfade_end", start))
|
||||
except (KeyError, TypeError, ValueError) as error:
|
||||
raise ValueError(
|
||||
f"FL prompt schedule section {index + 1} has invalid timing values."
|
||||
) from error
|
||||
prompt = value.get("prompt")
|
||||
curve = value.get("curve")
|
||||
if not all(math.isfinite(number) for number in (start, end, fade_in_end, fade_out_start)):
|
||||
if not all(math.isfinite(number) for number in (
|
||||
start,
|
||||
end,
|
||||
fade_in_end,
|
||||
fade_out_start,
|
||||
crossfade_start,
|
||||
crossfade_end,
|
||||
)):
|
||||
raise ValueError(f"FL prompt schedule section {index + 1} timing must be finite.")
|
||||
if not isinstance(prompt, str) or not prompt.strip():
|
||||
raise ValueError(f"FL prompt schedule section {index + 1} prompt is empty.")
|
||||
@@ -196,6 +217,10 @@ def _schedule_sections(schedule):
|
||||
raise ValueError(
|
||||
f"FL prompt schedule section {index + 1} has invalid fade boundaries."
|
||||
)
|
||||
if not 0 <= crossfade_start <= start + _EPS <= crossfade_end + _EPS <= end + _EPS:
|
||||
raise ValueError(
|
||||
f"FL prompt schedule section {index + 1} has invalid crossfade boundaries."
|
||||
)
|
||||
sections.append({
|
||||
"line": value.get("line", index + 1),
|
||||
"start": start,
|
||||
@@ -203,8 +228,24 @@ def _schedule_sections(schedule):
|
||||
"prompt": prompt.strip(),
|
||||
"fade_in_end": fade_in_end,
|
||||
"fade_out_start": fade_out_start,
|
||||
"crossfade_start": crossfade_start,
|
||||
"crossfade_end": crossfade_end,
|
||||
"curve": curve,
|
||||
})
|
||||
for index, section in enumerate(sections):
|
||||
if section["crossfade_end"] <= section["crossfade_start"] + _EPS:
|
||||
continue
|
||||
if index == 0:
|
||||
raise ValueError("FL prompt schedule first section cannot crossfade.")
|
||||
previous = sections[index - 1]
|
||||
if abs(previous["end"] - section["start"]) > _EPS:
|
||||
raise ValueError(
|
||||
f"FL prompt schedule section {index + 1} crossfade requires a touching previous section."
|
||||
)
|
||||
if section["crossfade_start"] < previous["start"] - _EPS:
|
||||
raise ValueError(
|
||||
f"FL prompt schedule section {index + 1} crossfade exceeds the previous section."
|
||||
)
|
||||
return sections
|
||||
|
||||
|
||||
@@ -402,6 +443,17 @@ def _curve(value, transition_mode):
|
||||
def _weights_at_time(sections, seconds, transition_mode, transition_seconds):
|
||||
weights = [0.0] * len(sections)
|
||||
if sections and "fade_in_end" in sections[0]:
|
||||
for index, (first, second) in enumerate(zip(sections, sections[1:])):
|
||||
crossfade_start = second.get("crossfade_start", second["start"])
|
||||
crossfade_end = second.get("crossfade_end", second["start"])
|
||||
if crossfade_start <= seconds < crossfade_end:
|
||||
amount = _curve(
|
||||
(seconds - crossfade_start) / (crossfade_end - crossfade_start),
|
||||
second["curve"],
|
||||
)
|
||||
weights[index] = 1.0 - amount
|
||||
weights[index + 1] = amount
|
||||
return weights
|
||||
for index, section in enumerate(sections):
|
||||
if not section["start"] <= seconds < section["end"]:
|
||||
continue
|
||||
@@ -556,6 +608,15 @@ def _merge_weights(weights, indices):
|
||||
]
|
||||
|
||||
|
||||
def _merge_section_weights(weights, indices):
|
||||
if not indices:
|
||||
return []
|
||||
return [
|
||||
min(1.0, sum(weights[index][position] for index in indices))
|
||||
for position in range(len(weights[0]))
|
||||
]
|
||||
|
||||
|
||||
def _h3_tensors(latent):
|
||||
samples = latent.get("samples")
|
||||
if not isinstance(samples, comfy.nested_tensor.NestedTensor):
|
||||
@@ -609,8 +670,8 @@ def _apply_timeline(timeline, latent):
|
||||
timeline["affect_audio"],
|
||||
)
|
||||
for group in timeline["conditioning_groups"]:
|
||||
group_video = _merge_weights(video_weights, group["section_indices"])
|
||||
group_audio = _merge_weights(audio_weights, group["section_indices"])
|
||||
group_video = _merge_section_weights(video_weights, group["section_indices"])
|
||||
group_audio = _merge_section_weights(audio_weights, group["section_indices"])
|
||||
mask = _flatten_mask(video.shape, audio.shape, group_video, group_audio)
|
||||
if torch.count_nonzero(mask):
|
||||
conditioning.extend(
|
||||
|
||||
@@ -65,7 +65,7 @@ class BeatPromptScheduleTests(unittest.TestCase):
|
||||
0.0,
|
||||
)
|
||||
|
||||
def test_node_returns_schedule_and_readable_preview(self):
|
||||
def test_node_returns_schedule_frames_audio_and_bpm(self):
|
||||
output = schedule.FL_Audio_Beat_Prompt_Schedule.execute(
|
||||
beat_positions=beat_json(),
|
||||
timeline="[0 - 48 | fade_in=6]\nSubject turns.",
|
||||
@@ -75,10 +75,11 @@ class BeatPromptScheduleTests(unittest.TestCase):
|
||||
).result
|
||||
|
||||
self.assertEqual(output[0]["type"], "fl_prompt_schedule")
|
||||
self.assertEqual(output[0]["version"], 1)
|
||||
self.assertEqual(output[0]["version"], 2)
|
||||
self.assertEqual(output[0]["source_unit"], "frames")
|
||||
self.assertIn("frames 0 - 48", output[1])
|
||||
self.assertIn("00:00.000 - 00:02.000", output[1])
|
||||
self.assertEqual(output[1], 60)
|
||||
self.assertIsNone(output[2])
|
||||
self.assertEqual(output[3], 120.0)
|
||||
|
||||
def test_seconds_mode_uses_direct_positions(self):
|
||||
output = schedule.FL_Audio_Beat_Prompt_Schedule.execute(
|
||||
@@ -116,8 +117,7 @@ class BeatPromptScheduleTests(unittest.TestCase):
|
||||
self.assertEqual(section["end"], 2.0)
|
||||
self.assertEqual(section["fade_in_end"], 0.25)
|
||||
self.assertEqual(section["fade_out_start"], 1.75)
|
||||
self.assertEqual(output.result[2], 2.0)
|
||||
self.assertEqual(output.result[3], 48)
|
||||
self.assertEqual(output.result[1], 48)
|
||||
|
||||
def test_frames_mode_rejects_fractional_frames(self):
|
||||
with self.assertRaisesRegex(ValueError, "whole frame"):
|
||||
@@ -132,19 +132,105 @@ class BeatPromptScheduleTests(unittest.TestCase):
|
||||
sequence_duration=0,
|
||||
)
|
||||
|
||||
def test_frame_length_rejects_sections_past_its_end(self):
|
||||
with self.assertRaisesRegex(ValueError, "beyond the sequence duration"):
|
||||
schedule.FL_Audio_Beat_Prompt_Schedule.execute(
|
||||
beat_positions=beat_json(),
|
||||
timeline="[0 - 2.1]\nCamera move.",
|
||||
default_fade_in=0.0,
|
||||
default_fade_out=0.0,
|
||||
curve="linear",
|
||||
time_unit="seconds",
|
||||
fps=24.0,
|
||||
sequence_duration=48,
|
||||
def test_crossfade_resolves_around_touching_frame_boundary(self):
|
||||
output = schedule.FL_Audio_Beat_Prompt_Schedule.execute(
|
||||
beat_positions=beat_json(),
|
||||
timeline=(
|
||||
"[0 - 24 | fade_out=6]\nFirst.\n"
|
||||
"[24 - 48 | fade_in=6 | crossfade=12]\nSecond."
|
||||
),
|
||||
default_fade_in=0.0,
|
||||
default_fade_out=0.0,
|
||||
curve="cosine",
|
||||
time_unit="frames",
|
||||
fps=24.0,
|
||||
sequence_duration=48,
|
||||
)
|
||||
|
||||
first, second = output.result[0]["sections"]
|
||||
self.assertEqual(first["fade_out_frames"], 0)
|
||||
self.assertEqual(second["fade_in_frames"], 0)
|
||||
self.assertEqual(second["crossfade_start_frame"], 18)
|
||||
self.assertEqual(second["crossfade_end_frame"], 30)
|
||||
self.assertEqual(second["crossfade_frames"], 12)
|
||||
self.assertEqual(second["crossfade_start"], 0.75)
|
||||
self.assertEqual(second["crossfade_end"], 1.25)
|
||||
|
||||
def test_crossfade_requires_touching_previous_section(self):
|
||||
with self.assertRaisesRegex(ValueError, "touching previous"):
|
||||
schedule._parse_schedule(
|
||||
"[0 - 24]\nFirst.\n[30 - 48 | crossfade=8]\nSecond.",
|
||||
0.0,
|
||||
0.0,
|
||||
"frames",
|
||||
)
|
||||
|
||||
def test_crossfade_cannot_exceed_shorter_adjacent_section(self):
|
||||
with self.assertRaisesRegex(ValueError, "shorter adjacent"):
|
||||
schedule._parse_schedule(
|
||||
"[0 - 8]\nFirst.\n[8 - 48 | crossfade=12]\nSecond.",
|
||||
0.0,
|
||||
0.0,
|
||||
"frames",
|
||||
)
|
||||
|
||||
def test_crossfade_must_be_a_whole_frame(self):
|
||||
with self.assertRaisesRegex(ValueError, "whole frame"):
|
||||
schedule._parse_schedule(
|
||||
"[0 - 24]\nFirst.\n[24 - 48 | crossfade=5.5]\nSecond.",
|
||||
0.0,
|
||||
0.0,
|
||||
"frames",
|
||||
)
|
||||
|
||||
def test_frame_length_crops_sections_at_its_end(self):
|
||||
output = schedule.FL_Audio_Beat_Prompt_Schedule.execute(
|
||||
beat_positions=beat_json(),
|
||||
timeline="[0 - 2.1]\nCamera move.",
|
||||
default_fade_in=0.0,
|
||||
default_fade_out=0.0,
|
||||
curve="linear",
|
||||
time_unit="seconds",
|
||||
fps=24.0,
|
||||
sequence_duration=48,
|
||||
)
|
||||
|
||||
section = output.result[0]["sections"][0]
|
||||
self.assertEqual(section["start"], 0.0)
|
||||
self.assertEqual(section["end"], 2.0)
|
||||
self.assertEqual(section["end_frame"], 48)
|
||||
|
||||
def test_frame_length_discards_sections_after_its_end(self):
|
||||
output = schedule.FL_Audio_Beat_Prompt_Schedule.execute(
|
||||
beat_positions=beat_json(),
|
||||
timeline="[49 - 59]\nCamera move.",
|
||||
default_fade_in=0.0,
|
||||
default_fade_out=0.0,
|
||||
curve="linear",
|
||||
time_unit="frames",
|
||||
fps=24.0,
|
||||
sequence_duration=48,
|
||||
)
|
||||
|
||||
self.assertEqual(output.result[0]["sections"], [])
|
||||
|
||||
def test_frame_length_preserves_visible_part_of_fade_out(self):
|
||||
output = schedule.FL_Audio_Beat_Prompt_Schedule.execute(
|
||||
beat_positions=beat_json(),
|
||||
timeline="[40 - 52 | fade_out=8]\nCamera move.",
|
||||
default_fade_in=0.0,
|
||||
default_fade_out=0.0,
|
||||
curve="linear",
|
||||
time_unit="frames",
|
||||
fps=24.0,
|
||||
sequence_duration=48,
|
||||
)
|
||||
|
||||
section = output.result[0]["sections"][0]
|
||||
self.assertEqual(section["start_frame"], 40)
|
||||
self.assertEqual(section["end_frame"], 48)
|
||||
self.assertEqual(section["fade_out_frames"], 4)
|
||||
|
||||
def test_length_is_always_a_frame_count(self):
|
||||
output = schedule.FL_Audio_Beat_Prompt_Schedule.execute(
|
||||
beat_positions=beat_json(),
|
||||
@@ -157,8 +243,7 @@ class BeatPromptScheduleTests(unittest.TestCase):
|
||||
sequence_duration=48,
|
||||
)
|
||||
|
||||
self.assertEqual(output.result[2], 2.0)
|
||||
self.assertEqual(output.result[3], 48)
|
||||
self.assertEqual(output.result[1], 48)
|
||||
|
||||
def test_ui_payload_contains_exact_beat_map(self):
|
||||
output = schedule.FL_Audio_Beat_Prompt_Schedule.execute(
|
||||
@@ -181,6 +266,9 @@ class BeatPromptScheduleTests(unittest.TestCase):
|
||||
"end_frame": 48,
|
||||
"fade_in_frames": 6,
|
||||
"fade_out_frames": 6,
|
||||
"crossfade_start_frame": 0,
|
||||
"crossfade_end_frame": 0,
|
||||
"crossfade_frames": 0,
|
||||
"prompt": "Camera move.",
|
||||
"curve": "linear",
|
||||
}])
|
||||
@@ -249,7 +337,7 @@ class BeatPromptScheduleTests(unittest.TestCase):
|
||||
self.assertEqual(inputs["default_fade_in"].display_name, "default fade-in (frames)")
|
||||
self.assertEqual(inputs["default_fade_out"].display_name, "default fade-out (frames)")
|
||||
|
||||
def test_schema_appends_audio_inputs_and_outputs_without_shifting_existing_contract(self):
|
||||
def test_schema_exposes_compact_outputs(self):
|
||||
schema = schedule.FL_Audio_Beat_Prompt_Schedule.define_schema()
|
||||
|
||||
self.assertEqual(
|
||||
@@ -281,16 +369,13 @@ class BeatPromptScheduleTests(unittest.TestCase):
|
||||
[output.display_name for output in schema.outputs],
|
||||
[
|
||||
"prompt_schedule",
|
||||
"preview",
|
||||
"duration_seconds",
|
||||
"total_frames",
|
||||
"audio",
|
||||
"beat_positions",
|
||||
"drum_times",
|
||||
"BPM",
|
||||
],
|
||||
)
|
||||
|
||||
def test_uploaded_audio_drives_schedule_and_ecosystem_outputs(self):
|
||||
def test_uploaded_audio_drives_schedule_audio_and_bpm_outputs(self):
|
||||
audio = {"waveform": torch.zeros(1, 2, 48000), "sample_rate": 48000}
|
||||
analysis = {
|
||||
"bpm": 120.0,
|
||||
@@ -326,10 +411,12 @@ class BeatPromptScheduleTests(unittest.TestCase):
|
||||
trim_start_frame=12,
|
||||
)
|
||||
|
||||
self.assertIs(output.result[4], audio)
|
||||
self.assertEqual(json.loads(output.result[5])["detected_beat_times"], [0.05, 0.52])
|
||||
self.assertEqual(json.loads(output.result[6])["kick_times"], [0.05])
|
||||
self.assertEqual(output.ui["fl_prompt_sequencer"][0]["source_start"], 0.5)
|
||||
self.assertIs(output.result[2], audio)
|
||||
self.assertEqual(output.result[3], 120.0)
|
||||
payload = output.ui["fl_prompt_sequencer"][0]
|
||||
self.assertEqual(payload["detected_beat_times"], [0.05, 0.52])
|
||||
self.assertEqual(payload["drum_times"]["kick_times"], [0.05])
|
||||
self.assertEqual(payload["source_start"], 0.5)
|
||||
|
||||
def test_external_beats_must_match_uploaded_crop(self):
|
||||
analysis = {
|
||||
@@ -365,15 +452,12 @@ class BeatPromptScheduleTests(unittest.TestCase):
|
||||
beat_offset_ms=100,
|
||||
)
|
||||
|
||||
effective = json.loads(output.result[5])
|
||||
payload = output.ui["fl_prompt_sequencer"][0]
|
||||
self.assertEqual(effective["base_beat_times"], [0.1, 0.6, 1.2, 1.9])
|
||||
self.assertEqual(effective["beat_times"], [0.2, 0.7, 1.3, 2.0])
|
||||
self.assertEqual(effective["beat_offset_ms"], 100)
|
||||
self.assertEqual(effective["grid_interval_seconds"], 0.6)
|
||||
self.assertEqual(payload["base_beat_times"], [0.1, 0.6, 1.2, 1.9])
|
||||
self.assertEqual(payload["beat_times"], [0.2, 0.7, 1.3, 2.0])
|
||||
self.assertEqual(payload["beat_offset_ms"], 100)
|
||||
self.assertEqual(payload["grid_interval_seconds"], 0.6)
|
||||
self.assertEqual(output.result[3], 120.0)
|
||||
|
||||
def test_scheduler_density_controls_external_grid_and_output(self):
|
||||
output = schedule.FL_Audio_Beat_Prompt_Schedule.execute(
|
||||
@@ -387,14 +471,12 @@ class BeatPromptScheduleTests(unittest.TestCase):
|
||||
beat_grid_density="every_2_beats",
|
||||
)
|
||||
|
||||
effective = json.loads(output.result[5])
|
||||
payload = output.ui["fl_prompt_sequencer"][0]
|
||||
self.assertEqual(effective["beat_times"], [0.1, 1.2])
|
||||
self.assertEqual(effective["base_beat_times"], [0.1, 0.6, 1.2, 1.9])
|
||||
self.assertEqual(effective["beat_grid_density"], "every_2_beats")
|
||||
self.assertEqual(effective["grid_bpm"], 50.0)
|
||||
self.assertEqual(payload["beat_times"], [0.1, 1.2])
|
||||
self.assertEqual(payload["base_beat_times"], [0.1, 0.6, 1.2, 1.9])
|
||||
self.assertEqual(payload["beat_grid_density"], "every_2_beats")
|
||||
self.assertEqual(payload["grid_bpm"], 50.0)
|
||||
self.assertEqual(output.result[3], 120.0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -151,6 +151,87 @@ class TimelineMaskTests(unittest.TestCase):
|
||||
0.5,
|
||||
)
|
||||
|
||||
def test_schedule_crossfade_blends_adjacent_prompts_with_unit_coverage(self):
|
||||
sections = [
|
||||
{
|
||||
"start": 0.0,
|
||||
"end": 1.0,
|
||||
"fade_in_end": 0.0,
|
||||
"fade_out_start": 1.0,
|
||||
"crossfade_start": 0.0,
|
||||
"crossfade_end": 0.0,
|
||||
"curve": "cosine",
|
||||
"prompt": "First.",
|
||||
},
|
||||
{
|
||||
"start": 1.0,
|
||||
"end": 2.0,
|
||||
"fade_in_end": 1.0,
|
||||
"fade_out_start": 2.0,
|
||||
"crossfade_start": 0.75,
|
||||
"crossfade_end": 1.25,
|
||||
"curve": "cosine",
|
||||
"prompt": "Second.",
|
||||
},
|
||||
]
|
||||
|
||||
for seconds in (0.75, 0.875, 1.0, 1.125, 1.249):
|
||||
first, second = timeline._weights_at_time(sections, seconds, "hard", 0.0)
|
||||
self.assertAlmostEqual(first + second, 1.0)
|
||||
self.assertEqual(timeline._weights_at_time(sections, 0.75, "hard", 0.0), [1.0, 0.0])
|
||||
midpoint = timeline._weights_at_time(sections, 1.0, "hard", 0.0)
|
||||
self.assertAlmostEqual(midpoint[0], 0.5)
|
||||
self.assertAlmostEqual(midpoint[1], 0.5)
|
||||
|
||||
def test_schedule_crossfade_can_cover_video_and_audio_tokens(self):
|
||||
sections = [
|
||||
{
|
||||
"start": 0.0,
|
||||
"end": 5 / 24,
|
||||
"fade_in_end": 0.0,
|
||||
"fade_out_start": 5 / 24,
|
||||
"crossfade_start": 0.0,
|
||||
"crossfade_end": 0.0,
|
||||
"curve": "linear",
|
||||
"prompt": "First.",
|
||||
},
|
||||
{
|
||||
"start": 5 / 24,
|
||||
"end": self.duration,
|
||||
"fade_in_end": 5 / 24,
|
||||
"fade_out_start": self.duration,
|
||||
"crossfade_start": 3 / 24,
|
||||
"crossfade_end": 7 / 24,
|
||||
"curve": "linear",
|
||||
"prompt": "Second.",
|
||||
},
|
||||
]
|
||||
|
||||
video, audio = timeline._temporal_weights(
|
||||
sections,
|
||||
video_t=7,
|
||||
audio_t=37,
|
||||
transition_mode="hard",
|
||||
transition_frames=0,
|
||||
affect_audio="video and audio",
|
||||
)
|
||||
|
||||
for first, second in zip(video[0], video[1]):
|
||||
self.assertAlmostEqual(first + second, 1.0)
|
||||
for first, second in zip(audio[0], audio[1]):
|
||||
self.assertAlmostEqual(first + second, 1.0)
|
||||
|
||||
def test_repeated_prompt_crossfade_keeps_full_group_weight(self):
|
||||
merged = timeline._merge_section_weights(
|
||||
[
|
||||
[1.0, 0.75, 0.5, 0.25, 0.0],
|
||||
[0.0, 0.25, 0.5, 0.75, 1.0],
|
||||
],
|
||||
[0, 1],
|
||||
)
|
||||
|
||||
self.assertEqual(merged, [1.0] * 5)
|
||||
|
||||
def test_prompt_envelope_is_averaged_on_h3_temporal_grid(self):
|
||||
envelope = {
|
||||
"prompt": "Pulse.",
|
||||
@@ -349,6 +430,42 @@ class TimelineNodeTests(unittest.TestCase):
|
||||
self.assertEqual(output[3]["sections"][0]["prompt"], "Beat-timed action.")
|
||||
self.assertEqual(output[3]["transition_frames"], 0)
|
||||
|
||||
def test_external_schedule_version_two_keeps_crossfade_boundaries(self):
|
||||
schedule = {
|
||||
"type": "fl_prompt_schedule",
|
||||
"version": 2,
|
||||
"duration": 2.0,
|
||||
"sections": [
|
||||
{
|
||||
"line": 1,
|
||||
"start": 0.0,
|
||||
"end": 1.0,
|
||||
"fade_in_end": 0.0,
|
||||
"fade_out_start": 1.0,
|
||||
"crossfade_start": 0.0,
|
||||
"crossfade_end": 0.0,
|
||||
"curve": "cosine",
|
||||
"prompt": "First.",
|
||||
},
|
||||
{
|
||||
"line": 3,
|
||||
"start": 1.0,
|
||||
"end": 2.0,
|
||||
"fade_in_end": 1.0,
|
||||
"fade_out_start": 2.0,
|
||||
"crossfade_start": 0.75,
|
||||
"crossfade_end": 1.25,
|
||||
"curve": "cosine",
|
||||
"prompt": "Second.",
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
sections = timeline._schedule_sections(schedule)
|
||||
|
||||
self.assertEqual(sections[1]["crossfade_start"], 0.75)
|
||||
self.assertEqual(sections[1]["crossfade_end"], 1.25)
|
||||
|
||||
def test_node_accepts_multiple_prompt_envelopes_and_deduplicates_prompts(self):
|
||||
class Clip:
|
||||
def tokenize(self, prompt, **kwargs):
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user