[docs] add InterleaveThinker review package
This commit is contained in:
Mac Lee
@@ -3074,3 +3074,39 @@ Next recommended integration step:
|
||||
- validation matrix covering service, run CLI, prompt-set eval, trace eval,
|
||||
SFT, critic GRPO, planner GRPO, and real-checkpoint smokes;
|
||||
- PR split recommendation and unresolved risks.
|
||||
|
||||
## Stage 13 Execution: Review Package
|
||||
|
||||
Status: completed locally, pending commit/push.
|
||||
|
||||
Goal:
|
||||
|
||||
- Add a stable review entrypoint outside the long exploration log.
|
||||
- Summarize public surfaces, validation evidence, PR split, and remaining risks
|
||||
for reviewers.
|
||||
|
||||
Implemented changes:
|
||||
|
||||
- Added `docs/design/interleave_thinker.md`.
|
||||
- Public inference/orchestration surfaces.
|
||||
- Public training/model/reward surfaces.
|
||||
- Architecture boundaries.
|
||||
- Representative Modal validation matrix.
|
||||
- Recommended PR stack.
|
||||
- Remaining risks and review checklist.
|
||||
- Added the page to the MkDocs Design nav in `mkdocs.yml`.
|
||||
|
||||
Validation completed:
|
||||
|
||||
- Local `git diff --check` passed.
|
||||
- Local `pre-commit run --files docs/design/interleave_thinker.md mkdocs.yml`
|
||||
passed:
|
||||
- codespell;
|
||||
- PyMarkdown;
|
||||
- filename check;
|
||||
- suggestion hook.
|
||||
- yapf, ruff, mypy, and actionlint had no files to check.
|
||||
- Local `python scripts/check_docs_links.py` failed on pre-existing generated
|
||||
examples links under `docs/examples/`, `docs/getting_started/`, `docs/inference/`,
|
||||
and `docs/training/`. No failures referenced the new
|
||||
`docs/design/interleave_thinker.md` page.
|
||||
|
||||
@@ -0,0 +1,173 @@
|
||||
# InterleaveThinker Integration Design
|
||||
|
||||
This page summarizes the InterleaveThinker integration branch for reviewers. It
|
||||
captures the public FastVideo surfaces, validation evidence, recommended PR
|
||||
split, and remaining risks. The detailed execution log remains in
|
||||
`.agents/exploration/interleavethinker-fastvideo-integration.md`.
|
||||
|
||||
## Scope
|
||||
|
||||
The branch adds FastVideo-native support for InterleaveThinker-style workflows:
|
||||
|
||||
- generator compatibility through a FastVideo-backed `/edit` API;
|
||||
- planner -> generator -> critic inference orchestration;
|
||||
- prompt-set and trace-level evaluation tools;
|
||||
- Qwen3-VL planner and critic model wrappers;
|
||||
- planner and critic SFT configs;
|
||||
- planner and critic GRPO configs with optional reference-policy KL;
|
||||
- Gemini and Nano Banana API wrappers for reward and image-generation backends.
|
||||
|
||||
It does not vendor InterleaveThinker, EasyR1, Verl, LLaMA-Factory, or training
|
||||
framework internals from those projects. FastVideo owns the public orchestration,
|
||||
model-wrapper, reward, and training-method boundaries.
|
||||
|
||||
## Public Surfaces
|
||||
|
||||
### Inference And Orchestration
|
||||
|
||||
| Surface | Purpose |
|
||||
|---------|---------|
|
||||
| `fastvideo.entrypoints.interleave` | Public package exports for orchestration, providers, prompt-set evaluation, trace serialization, and trace evaluation. |
|
||||
| `fastvideo interleave-serve` | Starts an InterleaveThinker-compatible FastVideo image edit service with `/edit`, `/generate`, and `/v1/interleave/edit` routes. |
|
||||
| `fastvideo interleave-run` | Runs one planner -> generator -> critic trace from a nested YAML config. |
|
||||
| `fastvideo interleave-eval` | Runs the same workflow over a JSONL/JSON/text prompt set and writes per-sample traces plus `summary.json`. |
|
||||
| `scripts/interleave_thinker/evaluate_traces.py` | Evaluates saved trace outputs and optionally writes an HTML report with final-image thumbnails. |
|
||||
|
||||
Examples live under `examples/interleave/`:
|
||||
|
||||
- `flux2_klein_interleave_serve.yaml`;
|
||||
- `interleave_run.yaml`;
|
||||
- `eval_prompts.jsonl`;
|
||||
- `interleave_single_prompt.py`.
|
||||
|
||||
### Training
|
||||
|
||||
| Surface | Purpose |
|
||||
|---------|---------|
|
||||
| `fastvideo.train.models.interleave_thinker.Qwen3VLActorBase` | Shared Transformers Qwen3-VL runtime for planner and critic actors. |
|
||||
| `InterleaveThinkerPlannerModel` | FastVideo `ModelBase` wrapper for `InterleaveThinker/InterleaveThinker-Planner-8B`. |
|
||||
| `InterleaveThinkerCriticModel` | FastVideo `ModelBase` wrapper for `InterleaveThinker/Critic-SFT-8B` and `InterleaveThinker/InterleaveThinker-Critic-8B`. |
|
||||
| `InterleaveThinkerSFTMethod` | Response-token supervised fine-tuning method for planner and critic actors. |
|
||||
| `InterleaveThinkerRLMethod` | Managed GRPO-style loop for planner and critic actors. |
|
||||
| `fastvideo.train.methods.rl.common.grpo` | Shared GRPO math helpers. |
|
||||
| `fastvideo.train.methods.rl.rewards.interleave_thinker` | Format, critic, and planner reward utilities. |
|
||||
| `fastvideo.train.methods.rl.rewards.interleave_api` | Gemini and Nano Banana API-backed reward/image wrappers. |
|
||||
|
||||
Training examples live under:
|
||||
|
||||
- `examples/train/configs/interleave_thinker/planner_sft_lora.yaml`;
|
||||
- `examples/train/configs/interleave_thinker/critic_sft_lora.yaml`;
|
||||
- `examples/train/configs/interleave_thinker/planner_smoke.yaml`;
|
||||
- `examples/train/configs/rl/interleave_thinker/critic_grpo.yaml`;
|
||||
- `examples/train/configs/rl/interleave_thinker/planner_grpo.yaml`.
|
||||
|
||||
## Architecture Boundaries
|
||||
|
||||
The integration is intentionally split into independent layers:
|
||||
|
||||
| Layer | Owner | Notes |
|
||||
|-------|-------|-------|
|
||||
| Planner and critic actors | `fastvideo/train/models/interleave_thinker/` | Wrap Transformers Qwen3-VL checkpoints. They are training actors, not diffusion pipeline components. |
|
||||
| RL/SFT algorithms | `fastvideo/train/methods/` | Own loss, reward aggregation, advantage computation, KL, and optimizer cadence. |
|
||||
| Rewards and API clients | `fastvideo/train/methods/rl/rewards/` | Testable offline reward aggregation is separate from network-backed Gemini/Nano Banana clients. |
|
||||
| Image generation/editing | `fastvideo/entrypoints/interleave/generator.py` | Presents a small image backend protocol for FastVideo, Nano Banana, and fake backends. |
|
||||
| Runtime orchestration | `fastvideo/entrypoints/interleave/` | Plans steps, calls generator/edit backends, calls critic providers, records traces. |
|
||||
| Evaluation | `fastvideo/entrypoints/interleave/evaluation.py` and `trace_eval.py` | Prompt-set execution and saved-trace reporting remain outside training methods. |
|
||||
|
||||
This keeps the Qwen actor implementation reusable by SFT, planner GRPO, critic
|
||||
GRPO, and inference providers without coupling those paths to a specific
|
||||
generator service.
|
||||
|
||||
## Validation Matrix
|
||||
|
||||
All GPU/model validation below ran on Modal L40S through
|
||||
`fastvideo/tests/modal/launch_l40s_job.py`.
|
||||
|
||||
| Area | Evidence | Modal app |
|
||||
|------|----------|-----------|
|
||||
| Initial compatibility service and orchestration tests | `5 passed, 14 warnings`; pre-commit passed. | `ap-cREtgFJyncsIfsXn1B5hLL` |
|
||||
| API-backed model/reward wrappers | `27 passed, 14 warnings`; final pre-commit passed. | `ap-QOKlzapm5bSAo3c21lprwv` |
|
||||
| Critic backend hardening | `30 passed, 14 warnings`; pre-commit passed. | `ap-DplMFq23YYfBx34e6TcsRc` |
|
||||
| Real critic checkpoint smoke | Loaded `InterleaveThinker/Critic-SFT-8B`; generated one rollout; printed `SMOKE_OK`. | `ap-hDxj5MhLgdnGq22mRLjgIK` |
|
||||
| FastVideo RL loop skeleton | `22 passed, 14 warnings`; pre-commit passed. | `ap-2Z2sH2UfhMoPmKolG0KY6t` |
|
||||
| Shared Qwen actor and planner wrapper | `36 passed, 14 warnings`; pre-commit passed. | `ap-ZapOKZPOmhyMZFxZ0X1fQm` |
|
||||
| Real planner checkpoint smoke | Loaded `InterleaveThinker/InterleaveThinker-Planner-8B`; parsed 3 steps; printed `PLANNER_SMOKE_OK`. | `ap-BzH7QxVXoc5XFXBah5cJ2H` |
|
||||
| Real critic refactor smoke | Loaded critic wrapper after Qwen base refactor; printed `CRITIC_REFACTOR_SMOKE_OK`. | `ap-NGxUDBNJFiU30Wef0yAQN1` |
|
||||
| Planner/critic provider adapters | `20 passed, 14 warnings`; pre-commit passed. | `ap-wfRX2DCt30DN903gETbDpj` |
|
||||
| Real provider loop smoke | Real planner and critic with fake generator; printed `INTERLEAVE_PROVIDER_REAL_LOOP_SMOKE_OK`. | `ap-ZABadeyKBuGVcfy67LqmXt` |
|
||||
| Native interleave-run CLI | `14 passed, 14 warnings`; pre-commit passed. | `ap-4lOvHURSxKqFmit8TqnhBe` |
|
||||
| Real FastVideo generator smoke | FastVideo generator produced a real image through the run CLI; printed `INTERLEAVE_RUN_FASTVIDEO_SMOKE_OK`. | `ap-Tz4VQKximS0bSWx6Fb9aFe` |
|
||||
| Dataset normalization | `17 passed, 14 warnings`; pre-commit passed. | `ap-ISuDU2lwc6Pl5NYDZnnBEb` |
|
||||
| Planner and critic SFT | `20 passed, 14 warnings`; final pre-commit passed. | `ap-1jmIczO3KwZoP3WtLYOIxc` |
|
||||
| Critic GRPO policy loss | Broad InterleaveThinker test set: `28 passed, 14 warnings`; pre-commit passed. | `ap-aYBz0F0ZiQ2nTGndudnGaH` |
|
||||
| Real critic RL smoke | Loaded LoRA critic student; generated rollouts; completed one GRPO update; printed `INTERLEAVE_CRITIC_RL_SMOKE_OK`. | `ap-eXMO3I81OcCyxj53XbPWj9` |
|
||||
| Reference-policy KL | `16 passed, 14 warnings`; real reference smoke printed `INTERLEAVE_CRITIC_RL_REFERENCE_SMOKE_OK`. | `ap-UQ38OTnymREO9bz0L1QzC5` |
|
||||
| Planner GRPO | `37 passed, 14 warnings`; real planner GRPO smoke printed `INTERLEAVE_PLANNER_RL_SMOKE_OK`. | `ap-PDBijC8opxsMiMU0Uc064A` |
|
||||
| Prompt-set evaluation workflow | `15 passed, 14 warnings`; pre-commit passed. | `ap-eeQpAgNQvQGi2H8MB0kJCU` |
|
||||
| Trace-level evaluation reports | `19 passed, 14 warnings`; pre-commit passed. | `ap-s7ewT9rDZSTdPNhyYrEYO7` |
|
||||
|
||||
## Recommended PR Stack
|
||||
|
||||
The branch is large and should be split before review:
|
||||
|
||||
1. **Inference service and orchestration shell**
|
||||
- `fastvideo/entrypoints/interleave/schema.py`;
|
||||
- generator backend protocol;
|
||||
- compatibility `/edit` service;
|
||||
- `interleave-run` CLI;
|
||||
- fake-backend tests.
|
||||
2. **Qwen3-VL actor wrappers**
|
||||
- shared Qwen actor base;
|
||||
- planner and critic wrappers;
|
||||
- data normalization helpers;
|
||||
- real checkpoint load smokes.
|
||||
3. **SFT path**
|
||||
- `InterleaveThinkerSFTMethod`;
|
||||
- planner and critic SFT configs;
|
||||
- response-token masking tests.
|
||||
4. **Reward and API backend path**
|
||||
- InterleaveThinker reward parser/scorers;
|
||||
- Gemini and Nano Banana wrappers;
|
||||
- fake-client tests.
|
||||
5. **GRPO path**
|
||||
- shared GRPO helpers;
|
||||
- `InterleaveThinkerRLMethod`;
|
||||
- critic GRPO, reference KL, planner GRPO;
|
||||
- real one-step LoRA smokes.
|
||||
6. **Evaluation and docs**
|
||||
- prompt-set runner;
|
||||
- trace evaluator and HTML report;
|
||||
- examples and design docs.
|
||||
|
||||
Each PR should keep the handoff updated until it lands or is superseded.
|
||||
|
||||
## Remaining Risks
|
||||
|
||||
- **Full 8B training memory:** Real one-step LoRA smokes passed. Full-parameter
|
||||
8B optimizer training and longer distributed runs still need dedicated
|
||||
hardware validation.
|
||||
- **Checkpoint/resume:** Configs include checkpoint settings, but planner/critic
|
||||
SFT and GRPO checkpoint/resume smokes are not yet recorded.
|
||||
- **Closed-source API drift:** Gemini and Nano Banana wrappers are unit-tested
|
||||
with fake clients. Live API outputs can change and should not be deterministic
|
||||
CI baselines.
|
||||
- **EasyR1 parity:** The FastVideo GRPO path matches the important objective
|
||||
pieces used here, but it is not a wholesale EasyR1/Verl port. Distributed
|
||||
rollout semantics and memory strategy should remain explicit in docs.
|
||||
- **Native Qwen3-VL port:** The branch uses Transformers Qwen3-VL wrappers. A
|
||||
FastVideo-native Qwen3-VL port should only be considered if conversion,
|
||||
performance, or distributed execution needs justify it.
|
||||
- **End-to-end real generator cost:** Real planner/critic and real FastVideo
|
||||
generator pieces have smoke coverage, but large prompt-set runs with all real
|
||||
components can be expensive and should be scheduled intentionally.
|
||||
|
||||
## Review Checklist
|
||||
|
||||
- Confirm no training code imports from the legacy `fastvideo/training/` stack.
|
||||
- Confirm API clients import optional dependencies lazily.
|
||||
- Confirm fake-provider tests cover planner, critic, generator, reward, and
|
||||
trace-evaluation behavior without credentials.
|
||||
- Confirm real-checkpoint smoke commands use pushed commits, not uncommitted
|
||||
Modal patches.
|
||||
- Confirm public YAML configs are parseable and clearly state credential,
|
||||
dataset, and hardware assumptions.
|
||||
@@ -161,6 +161,7 @@ nav:
|
||||
- Debugging: utilities/debugging.md
|
||||
- Design:
|
||||
- Overview: design/overview.md
|
||||
- InterleaveThinker Integration: design/interleave_thinker.md
|
||||
- Training Architecture: design/training_architecture.md
|
||||
- Server Contracts:
|
||||
- Overview: design/server_contracts/index.md
|
||||
|
||||
Reference in New Issue
Block a user