diff --git a/.agents/exploration/interleavethinker-fastvideo-integration.md b/.agents/exploration/interleavethinker-fastvideo-integration.md index cd0a2c72d..58c9eb169 100644 --- a/.agents/exploration/interleavethinker-fastvideo-integration.md +++ b/.agents/exploration/interleavethinker-fastvideo-integration.md @@ -3074,3 +3074,39 @@ Next recommended integration step: - validation matrix covering service, run CLI, prompt-set eval, trace eval, SFT, critic GRPO, planner GRPO, and real-checkpoint smokes; - PR split recommendation and unresolved risks. + +## Stage 13 Execution: Review Package + +Status: completed locally, pending commit/push. + +Goal: + +- Add a stable review entrypoint outside the long exploration log. +- Summarize public surfaces, validation evidence, PR split, and remaining risks + for reviewers. + +Implemented changes: + +- Added `docs/design/interleave_thinker.md`. + - Public inference/orchestration surfaces. + - Public training/model/reward surfaces. + - Architecture boundaries. + - Representative Modal validation matrix. + - Recommended PR stack. + - Remaining risks and review checklist. +- Added the page to the MkDocs Design nav in `mkdocs.yml`. + +Validation completed: + +- Local `git diff --check` passed. +- Local `pre-commit run --files docs/design/interleave_thinker.md mkdocs.yml` + passed: + - codespell; + - PyMarkdown; + - filename check; + - suggestion hook. + - yapf, ruff, mypy, and actionlint had no files to check. +- Local `python scripts/check_docs_links.py` failed on pre-existing generated + examples links under `docs/examples/`, `docs/getting_started/`, `docs/inference/`, + and `docs/training/`. No failures referenced the new + `docs/design/interleave_thinker.md` page. diff --git a/docs/design/interleave_thinker.md b/docs/design/interleave_thinker.md new file mode 100644 index 000000000..457dcd9bd --- /dev/null +++ b/docs/design/interleave_thinker.md @@ -0,0 +1,173 @@ +# InterleaveThinker Integration Design + +This page summarizes the InterleaveThinker integration branch for reviewers. It +captures the public FastVideo surfaces, validation evidence, recommended PR +split, and remaining risks. The detailed execution log remains in +`.agents/exploration/interleavethinker-fastvideo-integration.md`. + +## Scope + +The branch adds FastVideo-native support for InterleaveThinker-style workflows: + +- generator compatibility through a FastVideo-backed `/edit` API; +- planner -> generator -> critic inference orchestration; +- prompt-set and trace-level evaluation tools; +- Qwen3-VL planner and critic model wrappers; +- planner and critic SFT configs; +- planner and critic GRPO configs with optional reference-policy KL; +- Gemini and Nano Banana API wrappers for reward and image-generation backends. + +It does not vendor InterleaveThinker, EasyR1, Verl, LLaMA-Factory, or training +framework internals from those projects. FastVideo owns the public orchestration, +model-wrapper, reward, and training-method boundaries. + +## Public Surfaces + +### Inference And Orchestration + +| Surface | Purpose | +|---------|---------| +| `fastvideo.entrypoints.interleave` | Public package exports for orchestration, providers, prompt-set evaluation, trace serialization, and trace evaluation. | +| `fastvideo interleave-serve` | Starts an InterleaveThinker-compatible FastVideo image edit service with `/edit`, `/generate`, and `/v1/interleave/edit` routes. | +| `fastvideo interleave-run` | Runs one planner -> generator -> critic trace from a nested YAML config. | +| `fastvideo interleave-eval` | Runs the same workflow over a JSONL/JSON/text prompt set and writes per-sample traces plus `summary.json`. | +| `scripts/interleave_thinker/evaluate_traces.py` | Evaluates saved trace outputs and optionally writes an HTML report with final-image thumbnails. | + +Examples live under `examples/interleave/`: + +- `flux2_klein_interleave_serve.yaml`; +- `interleave_run.yaml`; +- `eval_prompts.jsonl`; +- `interleave_single_prompt.py`. + +### Training + +| Surface | Purpose | +|---------|---------| +| `fastvideo.train.models.interleave_thinker.Qwen3VLActorBase` | Shared Transformers Qwen3-VL runtime for planner and critic actors. | +| `InterleaveThinkerPlannerModel` | FastVideo `ModelBase` wrapper for `InterleaveThinker/InterleaveThinker-Planner-8B`. | +| `InterleaveThinkerCriticModel` | FastVideo `ModelBase` wrapper for `InterleaveThinker/Critic-SFT-8B` and `InterleaveThinker/InterleaveThinker-Critic-8B`. | +| `InterleaveThinkerSFTMethod` | Response-token supervised fine-tuning method for planner and critic actors. | +| `InterleaveThinkerRLMethod` | Managed GRPO-style loop for planner and critic actors. | +| `fastvideo.train.methods.rl.common.grpo` | Shared GRPO math helpers. | +| `fastvideo.train.methods.rl.rewards.interleave_thinker` | Format, critic, and planner reward utilities. | +| `fastvideo.train.methods.rl.rewards.interleave_api` | Gemini and Nano Banana API-backed reward/image wrappers. | + +Training examples live under: + +- `examples/train/configs/interleave_thinker/planner_sft_lora.yaml`; +- `examples/train/configs/interleave_thinker/critic_sft_lora.yaml`; +- `examples/train/configs/interleave_thinker/planner_smoke.yaml`; +- `examples/train/configs/rl/interleave_thinker/critic_grpo.yaml`; +- `examples/train/configs/rl/interleave_thinker/planner_grpo.yaml`. + +## Architecture Boundaries + +The integration is intentionally split into independent layers: + +| Layer | Owner | Notes | +|-------|-------|-------| +| Planner and critic actors | `fastvideo/train/models/interleave_thinker/` | Wrap Transformers Qwen3-VL checkpoints. They are training actors, not diffusion pipeline components. | +| RL/SFT algorithms | `fastvideo/train/methods/` | Own loss, reward aggregation, advantage computation, KL, and optimizer cadence. | +| Rewards and API clients | `fastvideo/train/methods/rl/rewards/` | Testable offline reward aggregation is separate from network-backed Gemini/Nano Banana clients. | +| Image generation/editing | `fastvideo/entrypoints/interleave/generator.py` | Presents a small image backend protocol for FastVideo, Nano Banana, and fake backends. | +| Runtime orchestration | `fastvideo/entrypoints/interleave/` | Plans steps, calls generator/edit backends, calls critic providers, records traces. | +| Evaluation | `fastvideo/entrypoints/interleave/evaluation.py` and `trace_eval.py` | Prompt-set execution and saved-trace reporting remain outside training methods. | + +This keeps the Qwen actor implementation reusable by SFT, planner GRPO, critic +GRPO, and inference providers without coupling those paths to a specific +generator service. + +## Validation Matrix + +All GPU/model validation below ran on Modal L40S through +`fastvideo/tests/modal/launch_l40s_job.py`. + +| Area | Evidence | Modal app | +|------|----------|-----------| +| Initial compatibility service and orchestration tests | `5 passed, 14 warnings`; pre-commit passed. | `ap-cREtgFJyncsIfsXn1B5hLL` | +| API-backed model/reward wrappers | `27 passed, 14 warnings`; final pre-commit passed. | `ap-QOKlzapm5bSAo3c21lprwv` | +| Critic backend hardening | `30 passed, 14 warnings`; pre-commit passed. | `ap-DplMFq23YYfBx34e6TcsRc` | +| Real critic checkpoint smoke | Loaded `InterleaveThinker/Critic-SFT-8B`; generated one rollout; printed `SMOKE_OK`. | `ap-hDxj5MhLgdnGq22mRLjgIK` | +| FastVideo RL loop skeleton | `22 passed, 14 warnings`; pre-commit passed. | `ap-2Z2sH2UfhMoPmKolG0KY6t` | +| Shared Qwen actor and planner wrapper | `36 passed, 14 warnings`; pre-commit passed. | `ap-ZapOKZPOmhyMZFxZ0X1fQm` | +| Real planner checkpoint smoke | Loaded `InterleaveThinker/InterleaveThinker-Planner-8B`; parsed 3 steps; printed `PLANNER_SMOKE_OK`. | `ap-BzH7QxVXoc5XFXBah5cJ2H` | +| Real critic refactor smoke | Loaded critic wrapper after Qwen base refactor; printed `CRITIC_REFACTOR_SMOKE_OK`. | `ap-NGxUDBNJFiU30Wef0yAQN1` | +| Planner/critic provider adapters | `20 passed, 14 warnings`; pre-commit passed. | `ap-wfRX2DCt30DN903gETbDpj` | +| Real provider loop smoke | Real planner and critic with fake generator; printed `INTERLEAVE_PROVIDER_REAL_LOOP_SMOKE_OK`. | `ap-ZABadeyKBuGVcfy67LqmXt` | +| Native interleave-run CLI | `14 passed, 14 warnings`; pre-commit passed. | `ap-4lOvHURSxKqFmit8TqnhBe` | +| Real FastVideo generator smoke | FastVideo generator produced a real image through the run CLI; printed `INTERLEAVE_RUN_FASTVIDEO_SMOKE_OK`. | `ap-Tz4VQKximS0bSWx6Fb9aFe` | +| Dataset normalization | `17 passed, 14 warnings`; pre-commit passed. | `ap-ISuDU2lwc6Pl5NYDZnnBEb` | +| Planner and critic SFT | `20 passed, 14 warnings`; final pre-commit passed. | `ap-1jmIczO3KwZoP3WtLYOIxc` | +| Critic GRPO policy loss | Broad InterleaveThinker test set: `28 passed, 14 warnings`; pre-commit passed. | `ap-aYBz0F0ZiQ2nTGndudnGaH` | +| Real critic RL smoke | Loaded LoRA critic student; generated rollouts; completed one GRPO update; printed `INTERLEAVE_CRITIC_RL_SMOKE_OK`. | `ap-eXMO3I81OcCyxj53XbPWj9` | +| Reference-policy KL | `16 passed, 14 warnings`; real reference smoke printed `INTERLEAVE_CRITIC_RL_REFERENCE_SMOKE_OK`. | `ap-UQ38OTnymREO9bz0L1QzC5` | +| Planner GRPO | `37 passed, 14 warnings`; real planner GRPO smoke printed `INTERLEAVE_PLANNER_RL_SMOKE_OK`. | `ap-PDBijC8opxsMiMU0Uc064A` | +| Prompt-set evaluation workflow | `15 passed, 14 warnings`; pre-commit passed. | `ap-eeQpAgNQvQGi2H8MB0kJCU` | +| Trace-level evaluation reports | `19 passed, 14 warnings`; pre-commit passed. | `ap-s7ewT9rDZSTdPNhyYrEYO7` | + +## Recommended PR Stack + +The branch is large and should be split before review: + +1. **Inference service and orchestration shell** + - `fastvideo/entrypoints/interleave/schema.py`; + - generator backend protocol; + - compatibility `/edit` service; + - `interleave-run` CLI; + - fake-backend tests. +2. **Qwen3-VL actor wrappers** + - shared Qwen actor base; + - planner and critic wrappers; + - data normalization helpers; + - real checkpoint load smokes. +3. **SFT path** + - `InterleaveThinkerSFTMethod`; + - planner and critic SFT configs; + - response-token masking tests. +4. **Reward and API backend path** + - InterleaveThinker reward parser/scorers; + - Gemini and Nano Banana wrappers; + - fake-client tests. +5. **GRPO path** + - shared GRPO helpers; + - `InterleaveThinkerRLMethod`; + - critic GRPO, reference KL, planner GRPO; + - real one-step LoRA smokes. +6. **Evaluation and docs** + - prompt-set runner; + - trace evaluator and HTML report; + - examples and design docs. + +Each PR should keep the handoff updated until it lands or is superseded. + +## Remaining Risks + +- **Full 8B training memory:** Real one-step LoRA smokes passed. Full-parameter + 8B optimizer training and longer distributed runs still need dedicated + hardware validation. +- **Checkpoint/resume:** Configs include checkpoint settings, but planner/critic + SFT and GRPO checkpoint/resume smokes are not yet recorded. +- **Closed-source API drift:** Gemini and Nano Banana wrappers are unit-tested + with fake clients. Live API outputs can change and should not be deterministic + CI baselines. +- **EasyR1 parity:** The FastVideo GRPO path matches the important objective + pieces used here, but it is not a wholesale EasyR1/Verl port. Distributed + rollout semantics and memory strategy should remain explicit in docs. +- **Native Qwen3-VL port:** The branch uses Transformers Qwen3-VL wrappers. A + FastVideo-native Qwen3-VL port should only be considered if conversion, + performance, or distributed execution needs justify it. +- **End-to-end real generator cost:** Real planner/critic and real FastVideo + generator pieces have smoke coverage, but large prompt-set runs with all real + components can be expensive and should be scheduled intentionally. + +## Review Checklist + +- Confirm no training code imports from the legacy `fastvideo/training/` stack. +- Confirm API clients import optional dependencies lazily. +- Confirm fake-provider tests cover planner, critic, generator, reward, and + trace-evaluation behavior without credentials. +- Confirm real-checkpoint smoke commands use pushed commits, not uncommitted + Modal patches. +- Confirm public YAML configs are parseable and clearly state credential, + dataset, and hardware assumptions. diff --git a/mkdocs.yml b/mkdocs.yml index 91ed9e202..89d83a88a 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -161,6 +161,7 @@ nav: - Debugging: utilities/debugging.md - Design: - Overview: design/overview.md + - InterleaveThinker Integration: design/interleave_thinker.md - Training Architecture: design/training_architecture.md - Server Contracts: - Overview: design/server_contracts/index.md