diff --git a/.agents/skills/seed-ssim-references/SKILL.md b/.agents/skills/seed-ssim-references/SKILL.md index f62421b42..c06f2a203 100644 --- a/.agents/skills/seed-ssim-references/SKILL.md +++ b/.agents/skills/seed-ssim-references/SKILL.md @@ -1,6 +1,6 @@ --- name: seed-ssim-references -description: Run a new (or updated) SSIM test on Modal, capture the generated videos, and upload them to the HF reference repo so the test can act as a regression guard. Use when a `fastvideo/tests/ssim/test_*.py` file has been added and has no reference video yet on `FastVideo/ssim-reference-videos`. +description: Seed HF reference videos for a single newly-added SSIM test. Runs the test on Modal L40S, downloads the generated mp4s via `modal volume get`, pauses for the user to eyeball quality, then uploads only that test's files to `FastVideo/ssim-reference-videos`. Use when a new `fastvideo/tests/ssim/test_*_similarity.py` has just been added and has no references on HF yet. --- # Seed SSIM Reference Videos @@ -8,205 +8,232 @@ description: Run a new (or updated) SSIM test on Modal, capture the generated vi ## Purpose A brand-new SSIM test in `fastvideo/tests/ssim/` fails forever until its -reference videos exist on the HF dataset (`FastVideo/ssim-reference-videos` by -default). This skill runs the test on Modal's L40S pool, pulls the generated -videos back locally, moves them into the `reference_videos/` layout, and -uploads to HF. After it runs, re-executing the same test on CI (or locally -with the references downloaded) should pass. +reference videos exist on the HF dataset (`FastVideo/ssim-reference-videos`). +This skill: -Use this skill when: -- A `test_*_similarity.py` file has just landed on `main` and the HF repo - still reports "Reference video folder does not exist" for it. -- An existing SSIM test's config changed (resolution, steps, prompt) and its - references need to be regenerated. +1. Runs the test on Modal's L40S pool to generate the videos. +2. Downloads them to the local repo via `modal volume get`. +3. Pauses so the user can eyeball the mp4s and confirm quality. +4. Uploads only the new test's files to HF, with a guard that refuses to + overwrite anything already present. -Do not use this skill for regular CI runs — once the references exist, normal -`pytest` is enough. +The skill is run **manually**, once per new test. Before invoking it, the user +has already sanity-tested the new test locally — it launches `VideoGenerator` +and writes an mp4 without crashing. The skill does not re-test locally; it +goes straight to Modal L40S (which is what CI uses). -## Prerequisites +## When to use -- `modal` CLI installed and authenticated (`modal token set ...`). -- HF API token with write access to `FastVideo/ssim-reference-videos`, exposed - via one of: `HF_API_KEY`, `HUGGINGFACE_HUB_TOKEN`, `HF_TOKEN`. -- The test file under review already: - - declares `REQUIRED_GPUS` at module scope (so the Modal orchestrator - schedules correctly); - - exposes `_MODEL_TO_PARAMS` (so CI can split one subprocess per - model id); - - registers in `fastvideo/tests/ssim/` using the shared helpers - (`resolve_inference_device_reference_folder`, - `run_text_to_video_similarity_test`, etc.). -- The `ghcr.io/hao-ai-lab/fastvideo/fastvideo-dev:` container - tag is pullable (set `IMAGE_VERSION` or accept the `latest` default). +- A new `test_*_similarity.py` file has been added in `fastvideo/tests/ssim/` + and the HF dataset has no `reference_videos/default/L40S_reference_videos//` + subtree for it yet. + +## When not to use + +- Regular CI runs — once refs exist, `pytest fastvideo/tests/ssim/` downloads + them automatically. +- Re-seeding an existing test. That requires `--force` on the upload step, and + is out of scope here; treat as a separate, deliberate operation. ## Inputs +The skill has **one required input**: the path to the new SSIM test file. +Prompt the user for it if they didn't supply it. + | Parameter | Required | Description | |-----------|----------|-------------| -| `test_file` | Yes | Path to the new test file, e.g. `fastvideo/tests/ssim/test_ltx2_similarity.py`. | -| `model_ids` | No | Comma-separated subset of the test's model ids. Defaults to all. | -| `quality_tier` | No | `default` (CI-friendly) or `full_quality`. Seed both if the test ships both. Default `default`. | -| `commit` | No | Git commit to run against. Defaults to local `HEAD`. | -| `git_repo` | No | Git repo URL. Defaults to `origin` remote of the local clone. | -| `reference_repo` | No | Target HF dataset repo. Defaults to `FastVideo/ssim-reference-videos`. | -| `device_folder` | No | Reference subfolder name, e.g. `L40S_reference_videos`. Defaults to `L40S_reference_videos` (the Modal runner's GPU). | +| `test_file` | Yes | e.g. `fastvideo/tests/ssim/test_ltx2_similarity.py`. The skill's first action is to ask for this if missing. | + +Everything else is fixed: + +- Modal runner GPU: **L40S** (hardcoded in `fastvideo/tests/modal/ssim_test.py`). +- Device folder: `L40S_reference_videos`. +- Quality tier: `default` (the tier CI runs). The `full_quality` tier is not + seeded by this skill. +- HF repo: `FastVideo/ssim-reference-videos` (dataset). +- Multi-model test files: all model ids in `*_MODEL_TO_PARAMS` are seeded + together; the Modal run produces one mp4 per (model, prompt, backend) and + the upload scopes by `--model-id`, looping if there is more than one. + +## Prerequisites + +The user has confirmed: + +- `modal` CLI authenticated. +- `HF_API_KEY` (or `HUGGINGFACE_HUB_TOKEN` / `HF_TOKEN`) exported with write + access to `FastVideo/ssim-reference-videos`. +- The test file runs locally end-to-end (generates an mp4; SSIM assertion + failure due to missing reference is expected and fine). + +Fail fast if the token env var is missing. ## Steps -### 1. Verify preconditions +### 1. Ask for the test file -- Confirm `test_file` exists and matches `test_*_similarity.py`. -- Grep for `REQUIRED_GPUS` and `*_MODEL_TO_PARAMS` in the file — if missing, - stop and direct the user to add them (see `fastvideo/tests/ssim/README.md` - "Adding a New SSIM Test"). -- Confirm `HF_API_KEY` (or equivalent) is exported. -- Confirm `modal token current` succeeds. +If the user didn't name one, ask: *"Which SSIM test file do you want to seed +references for? (e.g. `fastvideo/tests/ssim/test_ltx2_similarity.py`)"*. -### 2. Run the test on Modal with video export enabled +Validate: -Run from the repo root: +- Path exists and matches `fastvideo/tests/ssim/test_*_similarity.py`. +- File defines a `*_MODEL_TO_PARAMS` dict — grep it to extract the set of + model ids. Those ids drive step 5. + +If either check fails, stop and tell the user what's wrong. + +### 2. Run the test on Modal L40S + +Pick a subdir name so repeated runs don't collide: ```bash -bash .agents/skills/seed-ssim-references/scripts/seed_ssim.sh \ - --test-file fastvideo/tests/ssim/test_ltx2_similarity.py \ - --quality-tier default +SHORT_COMMIT=$(git rev-parse --short=12 HEAD) +TIMESTAMP=$(date -u +%Y%m%d_%H%M%S) +SUBDIR="${TIMESTAMP}_${SHORT_COMMIT}" ``` -The script is a thin wrapper that: - -1. Computes `git_repo` and `git_commit` if not supplied. -2. Invokes: - -``` - modal run fastvideo/tests/modal/ssim_test.py \ - --test-files= \ - --sync-generated-to-volume \ - --skip-reference-download \ - --no-fail-fast \ - [--full-quality] - ``` - -`--skip-reference-download` prevents the conftest from pre-pulling - (there are no references yet); `--no-fail-fast` lets generation finish - before `_assert_similarity` raises; `--sync-generated-to-volume` copies - the generated videos to the Modal volume `hf-model-weights`. -3. Captures the Modal volume path printed at the end of the run — it looks - like `ssim_generated_videos//_/generated_videos`. - -### 3. Pull the generated videos back - -Using the path printed in step 2: - -```bash -modal volume get hf-model-weights \ - ssim_generated_videos///generated_videos \ - ./generated_videos_modal/ -``` - -### 4. Copy into the `reference_videos/` layout - -```bash -python fastvideo/tests/ssim/reference_videos_cli.py copy-local \ - --quality-tier \ - --generated-dir ./generated_videos_modal//L40S_reference_videos \ - --device-folder L40S_reference_videos -``` - -This produces `fastvideo/tests/ssim/reference_videos//L40S_reference_videos///.mp4`. - -### 5. Upload to HF - -```bash -python fastvideo/tests/ssim/reference_videos_cli.py upload \ - --quality-tier \ - --device-folder L40S_reference_videos -``` - -The CLI reads the HF token from `HF_API_KEY` / `HUGGINGFACE_HUB_TOKEN` / -`HF_TOKEN` and fails fast if none are set. Override the target repo with -`FASTVIDEO_SSIM_REFERENCE_HF_REPO=` if you need a non-default -destination. - -### 6. Verify - -Re-run the test locally (or on Modal) **without** `--skip-reference-download`: +Then launch the Modal run: ```bash modal run fastvideo/tests/modal/ssim_test.py \ - --test-files=fastvideo/tests/ssim/test_ltx2_similarity.py + --git-repo="$(git config --get remote.origin.url)" \ + --git-commit="$(git rev-parse HEAD)" \ + --hf-api-key="$HF_API_KEY" \ + --test-files="" \ + --sync-generated-to-volume \ + --generated-volume-subdir="$SUBDIR" \ + --skip-reference-download \ + --no-fail-fast ``` -The conftest will now auto-download the references you just uploaded and the -SSIM comparison should pass. If it still fails, compare the local generated -video against the uploaded reference to see whether the threshold needs -adjusting. +Flag rationale: +- `--skip-reference-download`: no refs exist yet, so conftest must not try to + pull them. +- `--no-fail-fast`: lets the test finish generation before `_assert_similarity` + raises `FileNotFoundError: Reference video folder does not exist`. The + expected failure is what we want — the mp4 has already been written. +- `--sync-generated-to-volume` + `--generated-volume-subdir`: copies the + generated mp4s to the `hf-model-weights` Modal volume under + `ssim_generated_videos/default//generated_videos/` so we can pull + them locally. -### 7. Seed the full-quality tier if applicable +The Modal run will end with a nonzero exit (expected) and print a +`modal volume get hf-model-weights ssim_generated_videos/default//generated_videos ./generated_videos_modal/default` +command. Capture that `` — you need it for step 3. -If the test ships both tiers, repeat steps 2-5 with `--quality-tier -full_quality` (which passes `--full-quality` to the Modal run). +### 3. Download generated videos locally -## Outputs - -- Generated videos on the Modal volume `hf-model-weights` under - `ssim_generated_videos///generated_videos/...`. -- Local copies under `./generated_videos_modal//...`. -- Local references under `fastvideo/tests/ssim/reference_videos//L40S_reference_videos/...`. -- Uploaded references on HF at `` (default - `FastVideo/ssim-reference-videos`). - -## Example Usage - -Seed the new LTX-2 test (PR #1240) on main, default tier only: - -``` -Prompt: "Seed SSIM references for fastvideo/tests/ssim/test_ltx2_similarity.py" - test_file: fastvideo/tests/ssim/test_ltx2_similarity.py - quality_tier: default +```bash +modal volume get hf-model-weights \ + ssim_generated_videos/default/"$SUBDIR"/generated_videos \ + ./generated_videos_modal/default ``` -Seed both tiers for an existing test whose config just changed: +After this, the mp4s live at +`./generated_videos_modal/default/L40S_reference_videos///.mp4`. -``` -Prompt: "Re-seed test_wan_t2v_similarity.py; I bumped num_frames from 45 to 49" - test_file: fastvideo/tests/ssim/test_wan_t2v_similarity.py - quality_tier: default, then full_quality +### 4. PAUSE — user reviews quality + +Print the list of downloaded mp4s and their paths, then stop. Tell the user: + +> "Generated videos downloaded to `./generated_videos_modal/default/L40S_reference_videos/`. Please open them and confirm the quality looks correct. Reply **`upload`** to continue, or anything else to abort." + +Do not proceed until the user explicitly says `upload`. If they abort, leave +everything on disk so they can inspect further — no cleanup. + +### 5. Copy into the local reference layout + +Scoped copy — only the new test's mp4s. Loop over each `` extracted +in step 1: + +```bash +python fastvideo/tests/ssim/reference_videos_cli.py copy-local \ + --quality-tier default \ + --device-folder L40S_reference_videos \ + --generated-dir ./generated_videos_modal/default/L40S_reference_videos ``` -## Troubleshooting +(The `--generated-dir` points at the device-folder root; `copy-local` walks +all `//*.mp4` underneath it. Since the Modal run was scoped +to a single test file via `--test-files`, only that test's model(s) are +present — so the copy is implicitly per-test.) -- **"Hugging Face token is required"** — export `HF_API_KEY` in the shell - that runs `modal run`. Modal passes it through as a secret. -- **"No generated_videos directory found for quality tier"** — the test - didn't actually run inside the Modal container. Most common cause: - `REQUIRED_GPUS` larger than the partition has (the orchestrator skips - oversized tasks). Check the Modal logs under - `Partition : running / tasks`. -- **`ensure_reference_videos_available` still fails after upload** — the - HF dataset has caching; add a short sleep before step 6 or pass - `--skip-reference-download` and rely on local `reference_videos/` for - the first verification run. -- **Test passes locally but fails on Modal** — the Modal runner is L40S; - if you generated references on a different GPU SKU, use the right - `--device-folder` name (`A40_reference_videos`, `H100_reference_videos`, - etc.). +Result: `fastvideo/tests/ssim/reference_videos/default/L40S_reference_videos///.mp4`. + +### 6. Upload to HF — scoped per model_id, with overwrite guard + +For each ``: + +```bash +python fastvideo/tests/ssim/reference_videos_cli.py upload \ + --quality-tier default \ + --device-folder L40S_reference_videos \ + --model-id "" +``` + +The upload command: + +- Uploads **only** `reference_videos/default/L40S_reference_videos//`. +- **Refuses** if any file already exists at that path on HF (this is the + guard — seeding a new test should never clobber existing refs). To override, + the user must re-run with `--force`. If the guard fires, stop and report + exactly which files exist; do not silently `--force`. + +Reads the HF token from `HF_API_KEY` / `HUGGINGFACE_HUB_TOKEN` / `HF_TOKEN`. + +### 7. Report success + +List what was uploaded (paths in repo) and remind the user to push any +related code changes. Do **not** auto-verify by re-running Modal — the user +can run `pytest fastvideo/tests/ssim/` later to confirm end-to-end; +it will auto-download the refs they just uploaded. + +## Failure modes and how to handle them + +- **`HF_API_KEY` unset.** Stop before step 2. The Modal run needs it (passed + via `--hf-api-key`), and step 6 needs it for upload. +- **Modal run fails before generation.** No mp4s on the volume — nothing to + download. Fix the test locally (`pytest fastvideo/tests/ssim/`) + and retry from step 2. +- **`./generated_videos_modal/default/L40S_reference_videos/` missing after + `modal volume get`.** The run didn't produce videos (most likely the test + crashed before writing, or `REQUIRED_GPUS` exceeded the partition capacity + — see Modal logs). +- **Upload guard fires (files already exist).** The test name / model id + collides with something already on HF. Verify the user actually wants to + replace existing refs; if so, re-run the upload with `--force`. If not, + rename the model id in `*_MODEL_TO_PARAMS` and re-seed. +- **Quality looks wrong in step 4.** Abort. The mp4s stay on disk for + inspection. The fix is usually in the test's params (resolution, steps, + seed) — edit the test, then re-run the skill. + +## Design notes (for future skill maintainers) + +- The skill deliberately runs on Modal, **not** locally, because the CI + runner is L40S. Seeding from a different GPU SKU produces refs that CI's + L40S runs can't match (SSIM drifts across SKUs). +- The skill is default-tier only. `full_quality` refs are seeded by a + separate, deliberate operation — they double runtime and aren't what CI + gates on. +- The overwrite guard in `reference_videos_cli.py upload` is default-on + specifically because this skill exists. Re-seeding is a distinct operation + that requires explicit `--force`. ## References -- `fastvideo/tests/modal/ssim_test.py` (`run_ssim_tests` local entrypoint, - `run_ssim_partition` Modal function, `--sync-generated-to-volume` flow). -- `fastvideo/tests/ssim/reference_videos_cli.py` (`copy-local`, `upload`, - `download` subcommands, HF repo conventions). -- `fastvideo/tests/ssim/inference_similarity_utils.py` - (`run_text_to_video_similarity_test` + `_build_init_kwargs` — what each - test config ends up passing to `VideoGenerator.from_pretrained`). -- `fastvideo/tests/ssim/conftest.py` (`--skip-ssim-reference-download`, - `--ssim-full-quality`, `--ssim-reference-repo` pytest options). -- `fastvideo/tests/ssim/README.md` ("Adding a New SSIM Test" checklist, - reference videos layout, HF repo layout). +- `fastvideo/tests/modal/ssim_test.py` — Modal orchestrator; see + `--sync-generated-to-volume`, `--generated-volume-subdir`, + `--skip-reference-download`, `--no-fail-fast`. +- `fastvideo/tests/ssim/reference_videos_cli.py` — `copy-local`, `upload` + (with `--model-id`, `--force`), `download`, `ensure` subcommands. +- `fastvideo/tests/ssim/README.md` — reference layout, HF repo conventions. +- `fastvideo/tests/ssim/inference_similarity_utils.py` — + `run_text_to_video_similarity_test` + `_build_init_kwargs`: what each test + config passes to `VideoGenerator.from_pretrained`. ## Changelog | Date | Change | |------|--------| -| 2026-04-17 | Initial version. Documents the `--sync-generated-to-volume` seeding flow introduced by `ssim_test.py`. | +| 2026-04-17 | Initial version (Modal sync-to-volume flow). | +| 2026-04-21 | Rewrite: single-test scope, explicit user-review pause, per-`model_id` upload, HF overwrite guard. Dropped `scripts/seed_ssim.sh`. | diff --git a/.agents/skills/seed-ssim-references/scripts/seed_ssim.sh b/.agents/skills/seed-ssim-references/scripts/seed_ssim.sh deleted file mode 100755 index b578df480..000000000 --- a/.agents/skills/seed-ssim-references/scripts/seed_ssim.sh +++ /dev/null @@ -1,163 +0,0 @@ -#!/usr/bin/env bash -# Seed SSIM reference videos for a new test file. -# -# Runs the SSIM test on Modal with --sync-generated-to-volume, then prints the -# follow-up download / copy-local / upload commands so the operator can -# finish the seed. Designed to be safe to re-run: the Modal subdir is -# derived from the git commit + timestamp, so repeated runs don't overwrite -# each other. -# -# Usage: -# seed_ssim.sh --test-file fastvideo/tests/ssim/test_ltx2_similarity.py \ -# [--quality-tier default|full_quality] \ -# [--model-ids Model1,Model2] \ -# [--device-folder L40S_reference_videos] \ -# [--git-repo URL] [--git-commit SHA] - -set -euo pipefail - -TEST_FILE="" -QUALITY_TIER="default" -MODEL_IDS="" -DEVICE_FOLDER="L40S_reference_videos" -GIT_REPO="" -GIT_COMMIT="" - -usage() { - sed -n '1,/^$/p' "$0" | sed -e 's/^# \{0,1\}//' - exit "${1:-1}" -} - -while [[ $# -gt 0 ]]; do - case "$1" in - --test-file) - TEST_FILE="$2" - shift 2 - ;; - --quality-tier) - QUALITY_TIER="$2" - shift 2 - ;; - --model-ids) - MODEL_IDS="$2" - shift 2 - ;; - --device-folder) - DEVICE_FOLDER="$2" - shift 2 - ;; - --git-repo) - GIT_REPO="$2" - shift 2 - ;; - --git-commit) - GIT_COMMIT="$2" - shift 2 - ;; - -h|--help) - usage 0 - ;; - *) - echo "Unknown argument: $1" >&2 - usage 1 - ;; - esac -done - -if [[ -z "$TEST_FILE" ]]; then - echo "Missing --test-file" >&2 - usage 1 -fi -if [[ ! -f "$TEST_FILE" ]]; then - echo "Test file does not exist: $TEST_FILE" >&2 - exit 2 -fi - -case "$QUALITY_TIER" in - default|full_quality) ;; - *) - echo "quality_tier must be 'default' or 'full_quality', got: $QUALITY_TIER" >&2 - exit 2 - ;; -esac - -if ! grep -q '^REQUIRED_GPUS\s*=' "$TEST_FILE"; then - echo "Warning: $TEST_FILE has no REQUIRED_GPUS constant; Modal will schedule it as 1 GPU." >&2 -fi -if ! grep -q 'MODEL_TO_PARAMS' "$TEST_FILE"; then - echo "Warning: $TEST_FILE has no *_MODEL_TO_PARAMS dict; model_id splitting will be disabled." >&2 -fi - -HF_TOKEN="${HF_API_KEY:-${HUGGINGFACE_HUB_TOKEN:-${HF_TOKEN:-}}}" -if [[ -z "$HF_TOKEN" ]]; then - echo "Error: HF_API_KEY / HUGGINGFACE_HUB_TOKEN / HF_TOKEN not set." >&2 - echo " The upload step (step 5 in SKILL.md) will fail without it." >&2 - exit 3 -fi - -if [[ -z "$GIT_REPO" ]]; then - GIT_REPO="$(git config --get remote.origin.url)" -fi -if [[ -z "$GIT_COMMIT" ]]; then - GIT_COMMIT="$(git rev-parse HEAD)" -fi - -SHORT_COMMIT="${GIT_COMMIT:0:12}" -TIMESTAMP="$(date -u +%Y%m%d_%H%M%S)" -SUBDIR="${TIMESTAMP}_${SHORT_COMMIT}" -VOLUME_PATH="ssim_generated_videos/${QUALITY_TIER}/${SUBDIR}/generated_videos" -LOCAL_DOWNLOAD="./generated_videos_modal/${QUALITY_TIER}" - -FULL_QUALITY_FLAG="" -if [[ "$QUALITY_TIER" == "full_quality" ]]; then - FULL_QUALITY_FLAG="--full-quality" -fi - -MODEL_IDS_FLAG="" -if [[ -n "$MODEL_IDS" ]]; then - MODEL_IDS_FLAG="--model-ids=$MODEL_IDS" -fi - -echo "==========================================================" -echo "Seeding SSIM references:" -echo " test_file = $TEST_FILE" -echo " quality_tier = $QUALITY_TIER" -echo " git_commit = $GIT_COMMIT" -echo " volume_subdir = $SUBDIR" -echo " device_folder = $DEVICE_FOLDER" -echo "==========================================================" - -modal run fastvideo/tests/modal/ssim_test.py \ - --git-repo="$GIT_REPO" \ - --git-commit="$GIT_COMMIT" \ - --hf-api-key="$HF_TOKEN" \ - --test-files="$TEST_FILE" \ - $MODEL_IDS_FLAG \ - --sync-generated-to-volume \ - --generated-volume-subdir="$SUBDIR" \ - --skip-reference-download \ - --no-fail-fast \ - $FULL_QUALITY_FLAG - -echo -echo "Modal run finished. Next steps (copy into a shell with modal + HF creds):" -echo -echo " # 1. Pull the generated videos off the Modal volume" -echo " modal volume get hf-model-weights \\" -echo " $VOLUME_PATH \\" -echo " $LOCAL_DOWNLOAD" -echo -echo " # 2. Stage them into the reference_videos/ layout" -echo " python fastvideo/tests/ssim/reference_videos_cli.py copy-local \\" -echo " --quality-tier $QUALITY_TIER \\" -echo " --generated-dir $LOCAL_DOWNLOAD/$DEVICE_FOLDER \\" -echo " --device-folder $DEVICE_FOLDER" -echo -echo " # 3. Upload to HF (reads HF_API_KEY from env)" -echo " python fastvideo/tests/ssim/reference_videos_cli.py upload \\" -echo " --quality-tier $QUALITY_TIER \\" -echo " --device-folder $DEVICE_FOLDER" -echo -echo " # 4. Verify by re-running the test without --skip-reference-download" -echo " modal run fastvideo/tests/modal/ssim_test.py \\" -echo " --test-files=$TEST_FILE" diff --git a/fastvideo/tests/ssim/inference_similarity_utils.py b/fastvideo/tests/ssim/inference_similarity_utils.py index b40c878d6..57f7d6ee2 100644 --- a/fastvideo/tests/ssim/inference_similarity_utils.py +++ b/fastvideo/tests/ssim/inference_similarity_utils.py @@ -23,6 +23,7 @@ DEVICE_MAPPINGS = ( ("L40S", "L40S"), ("H100", "H100"), ("H200", "H200"), + ("B200", "B200"), ) diff --git a/fastvideo/tests/ssim/reference_videos_cli.py b/fastvideo/tests/ssim/reference_videos_cli.py index 3652cb715..81b2ee1f8 100644 --- a/fastvideo/tests/ssim/reference_videos_cli.py +++ b/fastvideo/tests/ssim/reference_videos_cli.py @@ -251,6 +251,8 @@ def upload_reference_videos( reference_dirs_by_tier: Sequence[tuple[str, Path]], token: str, private: bool, + model_id: str | None = None, + force: bool = False, ) -> None: HfApi, _ = _load_hf_sdk() api = HfApi(token=token) @@ -261,15 +263,44 @@ def upload_reference_videos( exist_ok=True, ) + try: + existing_repo_files = set( + api.list_repo_files(repo_id=repo_id, repo_type=repo_type)) + except Exception: + # Fresh repo or list failure — treat as empty so upload can proceed. + existing_repo_files = set() + for quality_tier, reference_dir in reference_dirs_by_tier: if not reference_dir.exists(): raise FileNotFoundError(f"Reference directory not found: {reference_dir}") - path_in_repo = f"{REFERENCE_VIDEOS_DIRNAME}/{quality_tier}/{reference_dir.name}" - print(f"Uploading {reference_dir.name} ({quality_tier}) to {repo_id} ...") + base_in_repo = f"{REFERENCE_VIDEOS_DIRNAME}/{quality_tier}/{reference_dir.name}" + if model_id: + folder_path = reference_dir / model_id + if not folder_path.exists(): + raise FileNotFoundError( + f"Model subfolder not found for upload: {folder_path}") + path_in_repo = f"{base_in_repo}/{model_id}" + else: + folder_path = reference_dir + path_in_repo = base_in_repo + + conflicts = sorted( + f for f in existing_repo_files + if f.startswith(f"{path_in_repo}/") or f == path_in_repo) + if conflicts and not force: + preview = "\n".join(f" - {c}" for c in conflicts[:10]) + more = f"\n ... and {len(conflicts) - 10} more" if len(conflicts) > 10 else "" + raise RuntimeError( + f"Refusing to overwrite existing HF files under {path_in_repo} " + f"({len(conflicts)} file(s) already present):\n{preview}{more}\n" + f"Re-run with --force to overwrite.") + + target_desc = f"{reference_dir.name}/{model_id}" if model_id else reference_dir.name + print(f"Uploading {target_desc} ({quality_tier}) to {repo_id}/{path_in_repo} ...") api.upload_folder( repo_id=repo_id, repo_type=repo_type, - folder_path=str(reference_dir), + folder_path=str(folder_path), path_in_repo=path_in_repo, token=token, ) @@ -500,6 +531,22 @@ def _build_parser() -> argparse.ArgumentParser: action="store_true", help="Create/use a private repo instead of public.", ) + upload_parser.add_argument( + "--model-id", + default=None, + help=( + "Restrict upload to a single model subfolder " + "(reference_videos///). " + "Use when seeding references for a single new test."), + ) + upload_parser.add_argument( + "--force", + action="store_true", + help=( + "Allow overwriting files that already exist at the target path on " + "Hugging Face. Off by default so seeding a new test cannot " + "clobber existing references."), + ) ensure_parser = subparsers.add_parser( "ensure", @@ -607,6 +654,8 @@ def main(argv: Sequence[str] | None = None) -> int: reference_dirs_by_tier=reference_dirs_by_tier, token=token, private=args.private, + model_id=args.model_id, + force=args.force, ) print("Upload complete.") return 0