Zhang Peiyuan
co-authored by
foreverpiano
foreverpiano
parent
8cfdf58a17
commit
8d41d505fe
@@ -1,21 +1,201 @@
|
||||
MIT License
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
Copyright (c) 2024 PKU-YUAN's Group (袁粒课题组-北大信工) and Rabbitpre AI
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
1. Definitions.
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright [2023] Lightning AI
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
@@ -1,108 +1,164 @@
|
||||
# Fast Video
|
||||
This is currently based on Open-Sora-1.2.0: https://github.com/PKU-YuanGroup/Open-Sora-Plan/tree/294993ca78bf65dec1c3b6fb25541432c545eda9
|
||||
# FastVideo
|
||||
|
||||
<div align="center">
|
||||
<a href=""><img src="https://img.shields.io/static/v1?label=API:H100&message=Replicate&color=pink"></a>  
|
||||
<a href=""><img src="https://img.shields.io/static/v1?label=Discuss&message=Discord&color=purple&logo=discord"></a>  
|
||||
</div>
|
||||
<br>
|
||||
<div align="center">
|
||||
<img src=assets/logo.png width="50%"/>
|
||||
</div>
|
||||
|
||||
As scaling laws from language models are applied to diffusion transformers, the number of parameters in diffusion models has grown significantly. This trend is even more pronounced in video models, where people are scaling not just the number of parameters but also the sequence length. As a result, traditional post-training workflows for diffusion models, such as fine-tuning, distillation, and inference, are becoming increasingly difficult to manage with frameworks like HF Diffusers, which are primarily designed for simple data-parallel workloads.
|
||||
|
||||
That is why we launched this FastVideo project to try to build a scalable framework for post-training various video diffusion models. As the tiny first step, we now provide a simple and efficient script to distill and finetune the 10B Mochi model. We will continue to add more features and models to this project in the future.
|
||||
|
||||
### Key Features
|
||||
|
||||
- FastMochi, a distilled Mochi model that can generate videos with merely 8 sampling steps.
|
||||
- Finetuning with FSDP (both master weight and ema weight), sequence parallelism, and selective gradient checkpointing.
|
||||
- LoRA coupled with pecomputed the latents and text embedding for minumum memory consumption.
|
||||
- Finetuning with both image and videos.
|
||||
|
||||
## Change Log
|
||||
|
||||
|
||||
- ```2024/12/06```: `FastMochi` v0.0.1 is released.
|
||||
|
||||
|
||||
## Fast and High-Quality Text-to-video Generation
|
||||
|
||||
### 8-Step Results of FastMochi
|
||||
|
||||
<table class="center">
|
||||
<td><img src=assets/8steps/1.gif width="320"></td></td>
|
||||
<td><img src=assets/8steps/2.gif width="320"></td></td></td>
|
||||
<tr>
|
||||
<td style="text-align:center;" width="320">tmp</td>
|
||||
<td style="text-align:center;" width="320">tmp</td>
|
||||
<tr>
|
||||
</table >
|
||||
|
||||
|
||||
## Table of Contents
|
||||
|
||||
Jump to a specific section:
|
||||
|
||||
- [🔧 Installation](#-installation)
|
||||
- [🚀 Inference](#-inference)
|
||||
- [🎯 Distill](#-distill)
|
||||
- [⚡ Finetune](#-lora-finetune)
|
||||
|
||||
|
||||
## 🔧 Installation
|
||||
|
||||
## Envrironment
|
||||
Change the index-url cuda version according to your system.
|
||||
```
|
||||
conda create -n fastvideo python=3.10.12
|
||||
conda activate fastvideo
|
||||
pip3 install torch==2.5.0 torchvision --index-url https://download.pytorch.org/whl/cu121
|
||||
pip install git+https://github.com/huggingface/diffusers.git@76b7d86a9a5c0c2186efa09c4a67b5f5666ac9e3
|
||||
conda create -n fastmochi python=3.10.0 -y && conda activate fastmochi
|
||||
pip3 install torch==2.5.0 torchvision --index-url https://download.pytorch.org/whl/cu121
|
||||
pip install packaging ninja && pip install flash-attn==2.7.0.post2 --no-build-isolation
|
||||
"git+https://github.com/huggingface/diffusers.git@bf64b32652a63a1865a0528a73a13652b201698b"
|
||||
git clone https://github.com/hao-ai-lab/FastVideo.git
|
||||
cd FastVideo && pip install -e .
|
||||
```
|
||||
|
||||
```
|
||||
pip install -e . && pip install -e ".[train]"
|
||||
sudo apt-get update && apt install screen && pip install watch gpustat
|
||||
|
||||
|
||||
|
||||
## 🚀 Inference
|
||||
|
||||
Use [scripts/download_hf.py](scripts/download_hf.py) to download the hugging-face style model to a local directory. Use it like this:
|
||||
```bash
|
||||
python scripts/download_hf.py --repo_id=FastVideo/FastMochi --local_dir=data/FastMochi --repo_type=model
|
||||
```
|
||||
|
||||
## Prepare Data & Models
|
||||
We've prepared some debug data to facilitate development. To make sure the training pipeline is correct, train on the debug data and make sure the model overfit on it (feed it the same text prompt and see if the output video is the same as the training data)
|
||||
|
||||
Start the gradio UI with
|
||||
```
|
||||
python3 ./demos/gradio_ui.py --model_dir weights/ --cpu_offload
|
||||
```
|
||||
We also provide CLI inference script featured with sequence parallelism.
|
||||
|
||||
```
|
||||
mkdir data && mkdir data/outputs/
|
||||
python scripts/download_hf.py --repo_id=Stealths-Video/mochi_diffuser --local_dir=data/mochi --repo_type=model
|
||||
python scripts/download_hf.py --repo_id=Stealths-Video/Merge-30k-Data --local_dir=data/Merge-30k-Data --repo_type=dataset
|
||||
export NUM_GPUS=4
|
||||
|
||||
torchrun --nnodes=1 --nproc_per_node=$NUM_GPUS \
|
||||
fastvideo/sample/sample_t2v_mochi.py \
|
||||
--model_path data/mochi \
|
||||
--prompt_path assets/prompt.txt \
|
||||
--num_frames 163 \
|
||||
--height 480 \
|
||||
--width 848 \
|
||||
--num_inference_steps 8 \
|
||||
--guidance_scale 4.5 \
|
||||
--output_path outputs_video/demo_video \
|
||||
--shift 8 \
|
||||
--seed 42 \
|
||||
--scheduler_type "pcm_linear_quadratic" \
|
||||
--linear_threshold 0.1 \
|
||||
--linear_range 0.75
|
||||
```
|
||||
|
||||
For the mochi style, simply following the scripts list in mochi repo.
|
||||
|
||||
```
|
||||
git clone https://github.com/genmoai/mochi.git
|
||||
cd mochi
|
||||
|
||||
# install env
|
||||
...
|
||||
|
||||
python3 ./demos/cli.py --model_dir weights/ --cpu_offload
|
||||
```
|
||||
|
||||
|
||||
## 🎯 Distill
|
||||
|
||||
## 💰Hardware requirement
|
||||
|
||||
- VRAM is required for both distill 10B mochi model
|
||||
|
||||
To launch distillation, you will first need to prepare data in the following formats
|
||||
|
||||
```bash
|
||||
asset/example_data
|
||||
├── AAA.txt
|
||||
├── AAA.png
|
||||
├── BCC.txt
|
||||
├── BCC.png
|
||||
├── ......
|
||||
├── CCC.txt
|
||||
└── CCC.png
|
||||
```
|
||||
|
||||
We provide a dataset example here. First download testing data. Use [scripts/download_hf.py](scripts/download_hf.py) to download the data to a local directory. Use it like this:
|
||||
```bash
|
||||
python scripts/download_hf.py --repo_id=Stealths-Video/Merge-425-Data --local_dir=data/Merge-425-Data --repo_type=dataset
|
||||
python scripts/download_hf.py --repo_id=Stealths-Video/validation_embeddings --local_dir=data/validation_embeddings --repo_type=dataset
|
||||
cd data/Merge-30k-Data
|
||||
cat Merged30K.tar.gz.part.* > Merged30K.tar.gz
|
||||
rm Merged30K.tar.gz.part.*
|
||||
tar --use-compress-program="pigz --processes 64" -xvf Merged30K.tar.gz
|
||||
mv ephemeral/hao.zhang/codefolder/FastVideo-OSP/data/Merged-30K-Data/* .
|
||||
rm -r ephemeral
|
||||
rm Merged30K.tar.gz
|
||||
cd ../..
|
||||
```
|
||||
|
||||
## Things Learned
|
||||
1. shift8 clear but got structural artifacts
|
||||
2. lq, 0.025 vague
|
||||
3. adv not really helpful
|
||||
4. shift8 euler steps 50 v.s. 100 very similar
|
||||
5. 为啥image不会越distill越炸
|
||||
6. EMA, 大batchsize, 1.5,2.5,3.5,4.5
|
||||
7. Must have schedule
|
||||
8. phase 1, 2 learning rate 5e-6不行
|
||||
Then the distillation can be launched by:
|
||||
|
||||
## Experiments
|
||||
Scripts are located at scripts/experiment_N.sh
|
||||
|
||||
1. pcm_linear_quadratic， euler_steps 50, 0.025
|
||||
2. pcm_linear_quadratic， euler_steps 50, 0.05
|
||||
3. shift 8, euler_steps 100
|
||||
4. shift 8, euler_steps 50
|
||||
5. shift 8, euler_steps 100, adv
|
||||
6. pcm_linear_quadratic， euler_steps 50, 0.025, adv
|
||||
7. pcm_linear_quadratic， euler_steps 50, 0.05, multiphase 125
|
||||
8. pcm_linear_quadratic， euler_steps 50, 0.1, linear_range 0.75
|
||||
9. pcm_linear_quadratic， euler_steps 50, 0.05, range 0.75
|
||||
10. pcm_linear_quadratic， euler_steps 50, 0.05, batchsize 32
|
||||
11. pcm_linear_quadratic， euler_steps 50, learning rate,1e-7
|
||||
12. shift1, euler_steps 50
|
||||
|
||||
13. pcm_linear_quadratic， euler_steps 50, 0.1, linear_range 0.75, phase 1
|
||||
14. 4.5 cfg, validation no cfg, pcm_linear_quadratic， euler_steps 50, 0.1, linear_range 0.75
|
||||
15. pcm_linear_quadratic， euler_steps 50, 0.15, linear_range 0.75
|
||||
16. pcm_linear_quadratic， euler_steps 50, 0.1, linear_range 0.75 ema 0.95, decay 0.0
|
||||
```
|
||||
bash scripts/distill_t2v.sh
|
||||
```
|
||||
|
||||
|
||||
17. no cfg, validation no cfg, pcm_linear_quadratic， euler_steps 50, 0.1, linear_range 0.75
|
||||
18. shift16, euler_steps 50
|
||||
## ⚡ Lora Finetune
|
||||
|
||||
19. 4step_infer_shift16_euler_50
|
||||
20. 4step_infer_shift12_euler_50
|
||||
21. 4step_infer_lq_euler_50_thresh0.1_lrg_0.75
|
||||
22. pcm_linear_quadratic， euler_steps 50, 0.1, linear_range 0.75, phase 1, lr 1e-7
|
||||
23. lq_euler_50_thres0.1_lrg_0.75_bs_64
|
||||
24. lq_euler_50_thres0.1_lrg_0.75_lr5e-7
|
||||
|
||||
## 💰Hardware requirement
|
||||
|
||||
- VRAM is required for both distill 10B mochi model
|
||||
|
||||
To launch finetuning, you will first need to prepare data in the following formats.
|
||||
|
||||
|
||||
|
||||
25. shift1_euler_50_0.75_phase1
|
||||
26. kill
|
||||
27. pcm_linear_quadratic， euler_steps 50, 0.1, linear_range 0.75, phase 1, ema 0.95, cfg 4.5
|
||||
Then the finetuning can be launched by:
|
||||
|
||||
28. lq_euler_50_thresh0.1_lrg_0.75_phase1_ema0.95
|
||||
29. lq_euler_50_thres0.1_lrg_0.75_phase_ema0.95_cfg7
|
||||
30. lq_euler_50_thresh0.1_lrg_0.75_phase1_ema0.98_cfg4.5
|
||||
31. lq_euler_50_thresh0.1_lrg_0.75_phase1_lr_3e-7
|
||||
32. lq_euler_50_thresh0.15_lrg_0.75_phase1_ema0.95_cfg4.5
|
||||
33. lq_euler_50_thres0.1_linear_range_0.75_repro
|
||||
34. lq_euler_50_thres0.1_lrg_0.75_reproduc
|
||||
```
|
||||
bash scripts/lora_finetune.sh
|
||||
```
|
||||
|
||||
35. pcm_linear_quadratic， euler_steps 50, 0.1, linear_range 0.75, phase 1, learning rate 5e-6
|
||||
36. pcm_linear_quadratic， euler_steps 50, 0.1, linear_range 0.75, phase 2, learning rate 1e-6
|
||||
37. pcm_linear_quadratic， euler_steps 50, 0.1, linear_range 0.75, phase 2, learning rate 5e-6
|
||||
38. lq_euler_50_thres0.1_linear_range_0.75, learning rate 5e-6
|
||||
39. lq_euler_50_thres0.1_linear_range_0.75, learning rate 1e-5
|
||||
40. pcm_linear_quadratic， euler_steps 50, 0.1, linear_range 0.75, phase 1, learning rate 1e-6
|
||||
|
||||
|
||||
41. lq_euler_50_thres0.1_lrg_0.75_reproduce
|
||||
42. pcm_linear_quadratic， euler_steps 50, 0.1, linear_range 0.75, phase 4, learning rate 1e-6
|
||||
43. pcm_linear_quadratic， euler_steps 50, 0.1, linear_range 0.75, phase 1, learning rate 1e-6, cfg 6.0
|
||||
44. lq_euler_50_thres0.1_lrg_0.75_phase1_lr_5e-6_test_norm
|
||||
45. lq_euler_50_thres0.1_lrg_0.75_phase1_lr_5e-6_pred_decay_0.1_latent14
|
||||
46. lq_euler_50_thres0.1_lrg_0.75_phase1_lr1e-6_pred_decay0.1
|
||||
47. lq_euler_50_thres0.1_lrg_0.75_phase1_lr1e-6_pred_decay0.05
|
||||
48. lq_euler_50_thres0.1_lrg_0.75_phase1_lr1e-6_pred_decay0.01
|
||||
## Acknowledgement
|
||||
We learned from and reused code from the following projects: [PCM](https://github.com/G-U-N/Phased-Consistency-Model), [diffusers](https://github.com/huggingface/diffusers), and [OpenSoraPlan](https://github.com/PKU-YuanGroup/Open-Sora-Plan).
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 1.3 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.3 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 380 KiB |
@@ -0,0 +1,9 @@
|
||||
A hand enters the frame, pulling a sheet of plastic wrap over three balls of dough placed on a wooden surface. The plastic wrap is stretched to cover the dough more securely. The hand adjusts the wrap, ensuring that it is tight and smooth over the dough. The scene focuses on the hand's movements as it secures the edges of the plastic wrap. No new objects appear, and the camera remains stationary, focusing on the action of covering the dough.
|
||||
A vintage train snakes through the mountains, its plume of white steam rising dramatically against the jagged peaks. The cars glint in the late afternoon sun, their deep crimson and gold accents lending a touch of elegance. The tracks carve a precarious path along the cliffside, revealing glimpses of a roaring river far below. Inside, passengers peer out the large windows, their faces lit with awe as the landscape unfolds.
|
||||
A crowded rooftop bar buzzes with energy, the city skyline twinkling like a field of stars in the background. Strings of fairy lights hang above, casting a warm, golden glow over the scene. Groups of people gather around high tables, their laughter blending with the soft rhythm of live jazz. The aroma of freshly mixed cocktails and charred appetizers wafts through the air, mingling with the cool night breeze.
|
||||
En "The Matrix", Neo, interpretado por Keanu Reeves, personifica la lucha contra un sistema opresor a través de su icónica imagen, que incluye unos anteojos oscuros. Estos lentes no son solo un accesorio de moda; representan una barrera entre la realidad y la percepción. Al usar estos anteojos, Neo se sumerge en un mundo donde la verdad se oculta detrás de ilusiones y engaños. La oscuridad de los lentes simboliza la ignorancia y el control que las máquinas tienen sobre la humanidad, mientras que su propia búsqueda de la verdad lo lleva a descubrir sus auténticos poderes. La escena en que se los pone se convierte en un momento crucial, marcando su transformación de un simple programador a "El Elegido". Esta imagen se ha convertido en un ícono cultural, encapsulando el mensaje de que, al enfrentar la oscuridad, podemos encontrar la luz que nos guía hacia la libertad. Así, los anteojos de Neo se convierten en un símbolo de resistencia y autoconocimiento en un mundo manipulado.
|
||||
Medium close up. Low-angle shot. A woman in a 1950s retro dress sits in a diner bathed in neon light, surrounded by classic decor and lively chatter. The camera starts with a medium shot of her sitting at the counter, then slowly zooms in as she blows a shiny pink bubblegum bubble. The bubble swells dramatically before popping with a soft, playful burst. The scene is vibrant and nostalgic, evoking the fun and carefree spirit of the 1950s.
|
||||
Will Smith eats noodles.
|
||||
A short clip of the blonde woman taking a sip from her whiskey glass, her eyes locking with the camera as she smirks playfully. The background shows a group of people laughing and enjoying the party, with vibrant neon signs illuminating the space. The shot is taken in a way that conveys the feeling of a tipsy, carefree night out. The camera then zooms in on her face as she winks, creating a cheeky, flirtatious vibe.
|
||||
A superintelligent humanoid robot waking up. The robot has a sleek metallic body with futuristic design features. Its glowing red eyes are the focal point, emanating a sharp, intense light as it powers on. The scene is set in a dimly lit, high-tech laboratory filled with glowing control panels, robotic arms, and holographic screens. The setting emphasizes advanced technology and an atmosphere of mystery. The ambiance is eerie and dramatic, highlighting the moment of awakening and the robot's immense intelligence. Photorealistic style with a cinematic, dark sci-fi aesthetic. Aspect ratio: 16:9 --v 6.1
|
||||
A chimpanzee lead vocalist singing into a microphone on stage. The camera zooms in to show him singing. There is a spotlight on him.
|
||||
@@ -34,9 +34,7 @@ class LatentDataset(Dataset):
|
||||
prompt_embed_file = self.data_anno[idx]["prompt_embed_path"]
|
||||
prompt_attention_mask_file = self.data_anno[idx]["prompt_attention_mask"]
|
||||
# load
|
||||
latent = torch.load(os.path.join(self.latent_dir, latent_file), map_location="cpu", weights_only=True)
|
||||
# TODO: Hack
|
||||
|
||||
latent = torch.load(os.path.join(self.latent_dir, latent_file), map_location="cpu", weights_only=True)
|
||||
latent = latent.squeeze(0)[:, -self.num_latent_t:]
|
||||
if random.random() < self.cfg_rate:
|
||||
prompt_embed = self.uncond_prompt_embed
|
||||
|
||||
@@ -13,15 +13,17 @@ from tqdm import tqdm
|
||||
from PIL import Image
|
||||
from accelerate.logging import get_logger
|
||||
from fastvideo.utils.dataset_utils import DecordInit
|
||||
from fastvideo.utils.utils import text_preprocessing
|
||||
import torchvision
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
def text_preprocessing(text, support_Chinese=True):
|
||||
# The exact text cleaning as was in the training stage:
|
||||
text = clean_caption(text, support_Chinese=support_Chinese)
|
||||
text = clean_caption(text, support_Chinese=support_Chinese)
|
||||
return text
|
||||
|
||||
class SingletonMeta(type):
|
||||
"""
|
||||
这是一个元类，用于创建单例类。
|
||||
"""
|
||||
_instances = {}
|
||||
|
||||
def __call__(cls, *args, **kwargs):
|
||||
|
||||
+3
-24
@@ -1,14 +1,10 @@
|
||||
import argparse
|
||||
from email.policy import strict
|
||||
import logging
|
||||
import math
|
||||
import os
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
from fastvideo.utils.parallel_states import initialize_sequence_parallel_state, \
|
||||
destroy_sequence_parallel_group, get_sequence_parallel_state, nccl_info
|
||||
from fastvideo.utils.communications import sp_parallel_dataloader_wrapper, broadcast
|
||||
from fastvideo.model.mochi_latents_utils import normalize_mochi_dit_input
|
||||
from fastvideo.models.mochi_hf.mochi_latents_utils import normalize_mochi_dit_input
|
||||
from fastvideo.utils.validation import log_validation
|
||||
import time
|
||||
from torch.utils.data import DataLoader
|
||||
@@ -19,7 +15,7 @@ from torch.distributed.fsdp import (
|
||||
StateDictType,
|
||||
FullStateDictConfig,
|
||||
)
|
||||
from fastvideo.model.pipeline_mochi import linear_quadratic_schedule
|
||||
from fastvideo.models.mochi_hf.pipeline_mochi import linear_quadratic_schedule
|
||||
import json
|
||||
from torch.utils.data.distributed import DistributedSampler
|
||||
from fastvideo.utils.dataset_utils import LengthGroupedSampler
|
||||
@@ -27,14 +23,13 @@ import wandb
|
||||
from accelerate.utils import set_seed
|
||||
from tqdm.auto import tqdm
|
||||
from fastvideo.fsdp_util import get_dit_fsdp_kwargs, apply_fsdp_checkpointing
|
||||
from diffusers.utils import convert_unet_state_dict_to_peft
|
||||
from diffusers import (
|
||||
FlowMatchEulerDiscreteScheduler,
|
||||
)
|
||||
from fastvideo.distill.solver import EulerSolver, extract_into_tensor
|
||||
from copy import deepcopy
|
||||
from diffusers.optimization import get_scheduler
|
||||
from fastvideo.model.modeling_mochi import MochiTransformer3DModel
|
||||
from fastvideo.models.mochi_hf.modeling_mochi import MochiTransformer3DModel
|
||||
from diffusers.utils import check_min_version
|
||||
from fastvideo.dataset.latent_datasets import LatentDataset, latent_collate_function
|
||||
import torch.distributed as dist
|
||||
@@ -49,21 +44,6 @@ check_min_version("0.31.0")
|
||||
import time
|
||||
from collections import deque
|
||||
|
||||
import sys
|
||||
import pdb
|
||||
#ForkedPdb().set_trace()
|
||||
class ForkedPdb(pdb.Pdb):
|
||||
"""A Pdb subclass that may be used
|
||||
from a forked multiprocessing child
|
||||
|
||||
"""
|
||||
def interaction(self, *args, **kwargs):
|
||||
_stdin = sys.stdin
|
||||
try:
|
||||
sys.stdin = open('/dev/stdin')
|
||||
pdb.Pdb.interaction(self, *args, **kwargs)
|
||||
finally:
|
||||
sys.stdin = _stdin
|
||||
|
||||
def main_print(content):
|
||||
if int(os.environ['LOCAL_RANK']) <= 0:
|
||||
@@ -542,7 +522,6 @@ if __name__ == "__main__":
|
||||
parser.add_argument("--pretrained_model_name_or_path", type=str)
|
||||
parser.add_argument("--dit_model_name_or_path", type=str)
|
||||
parser.add_argument("--cache_dir", type=str, default='./cache_dir')
|
||||
parser.add_argument('--enable_stable_fp32', action='store_true') # TODO
|
||||
|
||||
# diffusion setting
|
||||
parser.add_argument("--ema_decay", type=float, default=0.95)
|
||||
|
||||
@@ -8,7 +8,7 @@ from diffusers.configuration_utils import ConfigMixin, register_to_config
|
||||
from diffusers.utils import BaseOutput, logging
|
||||
from diffusers.utils.torch_utils import randn_tensor
|
||||
from diffusers.schedulers.scheduling_utils import SchedulerMixin
|
||||
from fastvideo.model.pipeline_mochi import linear_quadratic_schedule
|
||||
from fastvideo.models.mochi_hf.pipeline_mochi import linear_quadratic_schedule
|
||||
|
||||
logger = logging.get_logger(__name__) # pylint: disable=invalid-name
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ from pathlib import Path
|
||||
from fastvideo.utils.parallel_states import initialize_sequence_parallel_state, \
|
||||
destroy_sequence_parallel_group, get_sequence_parallel_state, nccl_info
|
||||
from fastvideo.utils.communications import sp_parallel_dataloader_wrapper, broadcast
|
||||
from fastvideo.model.mochi_latents_utils import normalize_mochi_dit_input
|
||||
from fastvideo.models.mochi_hf.mochi_latents_utils import normalize_mochi_dit_input
|
||||
from fastvideo.utils.validation import log_validation
|
||||
import time
|
||||
from torch.utils.data import DataLoader
|
||||
@@ -19,7 +19,7 @@ from torch.distributed.fsdp import (
|
||||
FullStateDictConfig,
|
||||
)
|
||||
|
||||
from fastvideo.model.pipeline_mochi import linear_quadratic_schedule
|
||||
from fastvideo.models.mochi_hf.pipeline_mochi import linear_quadratic_schedule
|
||||
import json
|
||||
from torch.utils.data.distributed import DistributedSampler
|
||||
from fastvideo.utils.dataset_utils import LengthGroupedSampler
|
||||
@@ -35,7 +35,7 @@ from fastvideo.distill.discriminator import Discriminator
|
||||
from fastvideo.distill.solver import EulerSolver, extract_into_tensor
|
||||
from copy import deepcopy
|
||||
from diffusers.optimization import get_scheduler
|
||||
from fastvideo.model.modeling_mochi import MochiTransformer3DModel
|
||||
from fastvideo.models.mochi_hf.modeling_mochi import MochiTransformer3DModel
|
||||
from diffusers.utils import check_min_version
|
||||
from fastvideo.dataset.latent_datasets import LatentDataset, latent_collate_function
|
||||
import torch.distributed as dist
|
||||
|
||||
@@ -17,7 +17,7 @@ from torch.distributed.fsdp import (
|
||||
# ShardedStateDictConfig, # un-flattened param but shards, usable by other parallel schemes.
|
||||
)
|
||||
|
||||
from fastvideo.model.modeling_mochi import MochiTransformerBlock
|
||||
from fastvideo.models.mochi_hf.modeling_mochi import MochiTransformerBlock
|
||||
|
||||
from functools import partial
|
||||
|
||||
|
||||
@@ -1,123 +0,0 @@
|
||||
|
||||
import torch
|
||||
from torch import nn
|
||||
import numpy as np
|
||||
from torch.nn.utils.parametrizations import spectral_norm
|
||||
import os
|
||||
class DummyDiscriminator(nn.Module):
|
||||
def __init__(self, dim_in, num_layers):
|
||||
super().__init__()
|
||||
self.layers = nn.ModuleList()
|
||||
for _ in range(num_layers):
|
||||
self.layers.append(nn.Linear(dim_in, 1))
|
||||
|
||||
def forward(self, features):
|
||||
logits = []
|
||||
for layer, feature in zip(self.layers, features):
|
||||
mean = feature.mean(dim=1)
|
||||
logits.append(layer(mean))
|
||||
return torch.cat(logits, dim=1)
|
||||
|
||||
|
||||
class ResidualBlock(nn.Module):
|
||||
def __init__(self, fn):
|
||||
super().__init__()
|
||||
self.fn = fn
|
||||
|
||||
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
||||
return (self.fn(x) + x) / np.sqrt(2)
|
||||
|
||||
|
||||
class SpectralConv1d(nn.Module):
|
||||
def __init__(self, *args, **kwargs):
|
||||
super().__init__()
|
||||
self.conv = spectral_norm(nn.Conv1d(*args, **kwargs))
|
||||
def forward(self, x):
|
||||
return self.conv(x)
|
||||
|
||||
class BatchNormLocal(nn.Module):
|
||||
def __init__(self, num_features: int, affine: bool = True, virtual_bs: int = 8, eps: float = 1e-5):
|
||||
super().__init__()
|
||||
self.virtual_bs = virtual_bs
|
||||
self.eps = eps
|
||||
self.affine = affine
|
||||
|
||||
if self.affine:
|
||||
self.weight = nn.Parameter(torch.ones(num_features))
|
||||
self.bias = nn.Parameter(torch.zeros(num_features))
|
||||
|
||||
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
||||
shape = x.size()
|
||||
|
||||
# Calculate stats.
|
||||
mean = x.mean([0, 2], keepdim=True)
|
||||
var = x.var([0, 2], keepdim=True, unbiased=False)
|
||||
x = (x - mean) / (torch.sqrt(var + self.eps))
|
||||
|
||||
if self.affine:
|
||||
x = x * self.weight[None, :, None] + self.bias[None, :, None]
|
||||
|
||||
return x.view(shape)
|
||||
|
||||
def make_block(channels: int, kernel_size: int) -> nn.Module:
|
||||
return nn.Sequential(
|
||||
SpectralConv1d(
|
||||
channels,
|
||||
channels,
|
||||
kernel_size = kernel_size,
|
||||
padding = kernel_size//2,
|
||||
padding_mode = 'circular',
|
||||
),
|
||||
BatchNormLocal(channels),
|
||||
nn.LeakyReLU(0.2, True),
|
||||
)
|
||||
|
||||
class DiscHead(nn.Module):
|
||||
def __init__(self, feature_dim: int, text_c_dim: int, cmap_dim: int = 64, cnn_dim=512):
|
||||
super().__init__()
|
||||
self.channels = feature_dim
|
||||
self.text_c_dim = text_c_dim
|
||||
self.cmap_dim = cmap_dim
|
||||
self.down_proj = SpectralConv1d(feature_dim, cnn_dim, kernel_size=1, padding=0)
|
||||
self.main = nn.Sequential(
|
||||
make_block(cnn_dim, kernel_size=1),
|
||||
ResidualBlock(make_block(cnn_dim, kernel_size=9))
|
||||
)
|
||||
|
||||
self.cmapper = nn.Linear(self.text_c_dim, cmap_dim)
|
||||
self.cls = SpectralConv1d(cnn_dim, cmap_dim, kernel_size=1, padding=0)
|
||||
|
||||
def forward(self, x: torch.Tensor, c: torch.Tensor) -> torch.Tensor:
|
||||
h = self.down_proj(x)
|
||||
h = self.main(h)
|
||||
out = self.cls(h)
|
||||
|
||||
cmap = self.cmapper(c).unsqueeze(-1)
|
||||
out = (out * cmap).sum(1, keepdim=True) * (1 / np.sqrt(self.cmap_dim))
|
||||
|
||||
return out
|
||||
|
||||
|
||||
class LADDDiscriminator(nn.Module):
|
||||
def __init__(self, feature_dim, text_cond_dim, num_layers, layers_stride):
|
||||
super().__init__()
|
||||
heads = []
|
||||
for i in range(0, num_layers, layers_stride):
|
||||
heads.append(DiscHead(feature_dim, text_cond_dim))
|
||||
self.heads = nn.ModuleList(heads)
|
||||
self.layers_stride = layers_stride
|
||||
self.num_layers = num_layers
|
||||
|
||||
def forward(self, features, text_conditions) -> torch.Tensor:
|
||||
text_conditions = text_conditions.mean(1)
|
||||
# layer, B, L, C -> layer, B, C, L
|
||||
features = features.transpose(2, 3)
|
||||
logits = []
|
||||
for i in range(0, self.num_layers, self.layers_stride):
|
||||
head = self.heads[i//self.layers_stride]
|
||||
feat = features[i]
|
||||
logits.append(head(feat, text_conditions).view(feat.size(0), -1))
|
||||
logits = torch.cat(logits, dim=1)
|
||||
|
||||
|
||||
return logits
|
||||
@@ -1,155 +0,0 @@
|
||||
import torch
|
||||
from fastvideo.model.pipeline_mochi import MochiPipeline
|
||||
import torch.distributed as dist
|
||||
|
||||
from diffusers.utils import export_to_video
|
||||
from fastvideo.utils.parallel_states import initialize_sequence_parallel_state, nccl_info
|
||||
import argparse
|
||||
import os
|
||||
from diffusers.models.transformers.transformer_mochi import MochiTransformerBlock
|
||||
from fastvideo.model.modeling_mochi import MochiTransformer3DModel
|
||||
|
||||
|
||||
|
||||
import sys
|
||||
import pdb
|
||||
|
||||
class ForkedPdb(pdb.Pdb):
|
||||
"""
|
||||
PDB Subclass for debugging multi-processed code
|
||||
Suggested in: https://stackoverflow.com/questions/4716533/how-to-attach-debugger-to-a-python-subproccess
|
||||
"""
|
||||
def interaction(self, *args, **kwargs):
|
||||
_stdin = sys.stdin
|
||||
try:
|
||||
sys.stdin = open('/dev/stdin')
|
||||
pdb.Pdb.interaction(self, *args, **kwargs)
|
||||
finally:
|
||||
sys.stdin = _stdin
|
||||
def assert_all_close_list(input_list):
|
||||
for i in range(len(input_list) - 1):
|
||||
assert torch.allclose(input_list[i], input_list[i + 1]), f"input_list[{i}]: {input_list[i]}, input_list[{i+1}]: {input_list[i+1]}"
|
||||
|
||||
weight_dtype = torch.float32
|
||||
def initialize_distributed():
|
||||
local_rank = int(os.getenv('RANK', 0))
|
||||
world_size = int(os.getenv('WORLD_SIZE', 1))
|
||||
print('world_size', world_size)
|
||||
torch.cuda.set_device(local_rank)
|
||||
dist.init_process_group(backend='nccl', init_method='env://', world_size=world_size, rank=local_rank)
|
||||
return world_size
|
||||
|
||||
def main_print(content):
|
||||
if int(os.getenv('RANK', 0)) <= 0:
|
||||
print(content)
|
||||
|
||||
@torch.inference_mode
|
||||
def test_single_block(batch_size, device, seed):
|
||||
# set manual seed
|
||||
torch.manual_seed(seed)
|
||||
device = torch.cuda.current_device()
|
||||
block = MochiTransformerBlock(
|
||||
dim=768,
|
||||
num_attention_heads=12,
|
||||
attention_head_dim=64,
|
||||
pooled_projection_dim=256,
|
||||
qk_norm="rms_norm",
|
||||
activation_fn="swiglu",
|
||||
context_pre_only=False,
|
||||
).to(device)
|
||||
hidden_states = torch.randn(1, 16, 768).to(device).repeat(batch_size, 1, 1)
|
||||
encoder_hidden_states = torch.randn(1, 4, 256).to(device).repeat(batch_size, 1, 1)
|
||||
temb = torch.randn(1, 768).to(device).repeat(batch_size, 1)
|
||||
# shard hiddent_states according to world_size
|
||||
local_seq_length = hidden_states.shape[1] // nccl_info.sp_size
|
||||
hidden_states = hidden_states.narrow(1, nccl_info.global_rank * local_seq_length, local_seq_length)
|
||||
main_print(hidden_states.shape)
|
||||
hidden_states, encoder_hidden_states = block(
|
||||
hidden_states=hidden_states,
|
||||
encoder_hidden_states=encoder_hidden_states,
|
||||
temb=temb,
|
||||
)
|
||||
mean = hidden_states[0].mean()
|
||||
torch.distributed.all_reduce(mean, op=torch.distributed.ReduceOp.SUM)
|
||||
mean = mean / nccl_info.sp_size
|
||||
return mean
|
||||
|
||||
@torch.inference_mode
|
||||
def test_DiT(batch_size, transformer, seed):
|
||||
generator = torch.Generator(torch.cuda.current_device()).manual_seed(seed)
|
||||
device = torch.cuda.current_device()
|
||||
latent = torch.randn((1, 12, 8, 12, 8), device=device, dtype=weight_dtype, generator=generator).repeat(batch_size, 1, 1, 1, 1)
|
||||
prompt_embeds = torch.randn((1, 20, 4096), device=device, dtype=weight_dtype, generator=generator).repeat(batch_size, 1, 1)
|
||||
prompt_attention_mask = torch.ones((1, 20), device=device, dtype=weight_dtype).repeat(batch_size, 1)
|
||||
timestep = 0
|
||||
timestep = torch.tensor(timestep, device=device, dtype=weight_dtype).unsqueeze(0).repeat(batch_size)
|
||||
local_seq_length = latent.shape[2] // nccl_info.sp_size
|
||||
latent = latent.narrow(2, nccl_info.global_rank * local_seq_length, local_seq_length)
|
||||
# main_print(latent.shape)
|
||||
hidden_states = transformer(
|
||||
hidden_states=latent,
|
||||
encoder_hidden_states=prompt_embeds,
|
||||
encoder_attention_mask=prompt_attention_mask,
|
||||
timestep=timestep,
|
||||
return_dict=False,
|
||||
)[0]
|
||||
def calculate_mean(states):
|
||||
mean = states.mean()
|
||||
torch.distributed.all_reduce(mean, op=torch.distributed.ReduceOp.SUM)
|
||||
mean = mean / int(os.getenv('WORLD_SIZE', 1))
|
||||
return mean
|
||||
mean1 = calculate_mean(hidden_states[0])
|
||||
main_print(hidden_states.shape)
|
||||
if hidden_states.shape[0] > 1:
|
||||
mean2 = calculate_mean(hidden_states[1])
|
||||
return mean1, mean2
|
||||
return mean1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
world_size = initialize_distributed()
|
||||
device = torch.cuda.current_device()
|
||||
|
||||
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--seed", type=int, default=42)
|
||||
parser.add_argument("--test_single_block", action="store_true")
|
||||
args = parser.parse_args()
|
||||
seed = args.seed
|
||||
|
||||
if args.test_single_block:
|
||||
pass
|
||||
single_no_patch_bs_1 = test_single_block(1)
|
||||
single_no_patch_bs_2 = test_single_block(2)
|
||||
# check all close
|
||||
assert torch.allclose(single_no_patch_bs_1, single_no_patch_bs_2)
|
||||
single_patch_bs_1 = test_single_block(1)
|
||||
single_patch_bs_2 = test_single_block(2)
|
||||
assert torch.allclose(single_patch_bs_1, single_patch_bs_2)
|
||||
assert torch.allclose(single_no_patch_bs_1, single_patch_bs_2)
|
||||
initialize_sequence_parallel_state(world_size)
|
||||
|
||||
sp_patch_bs_1 = test_single_block(1)
|
||||
sp_patch_bs_2 = test_single_block(2)
|
||||
|
||||
assert torch.allclose(sp_patch_bs_1, sp_patch_bs_2)
|
||||
assert torch.allclose(single_no_patch_bs_1, sp_patch_bs_2)
|
||||
|
||||
else:
|
||||
transformer = MochiTransformer3DModel.from_pretrained("data/mochi/transformer", torch_dtype=weight_dtype).to(device)
|
||||
single_no_patch_bs_1 = test_DiT(1, transformer, seed)
|
||||
single_no_patch_bs_2_a, single_no_patch_bs_2_b = test_DiT(2, transformer, seed)
|
||||
|
||||
|
||||
single_patch_bs_1 = test_DiT(1, transformer, seed)
|
||||
single_patch_bs_2_a, single_patch_bs_2_b = test_DiT(2, transformer, seed)
|
||||
|
||||
initialize_sequence_parallel_state(world_size)
|
||||
|
||||
sp_patch_bs_1 = test_DiT(1, transformer, seed)
|
||||
sp_patch_bs_2_a, sp_patch_bs_2_b = test_DiT(2, transformer, seed)
|
||||
|
||||
assert_all_close_list([single_no_patch_bs_1, single_no_patch_bs_2_a, single_no_patch_bs_2_b, single_patch_bs_1, single_patch_bs_2_a, single_patch_bs_2_b, sp_patch_bs_1, sp_patch_bs_2_a, sp_patch_bs_2_b])
|
||||
|
||||
main_print(sp_patch_bs_1)
|
||||
|
||||
@@ -27,7 +27,7 @@ from diffusers.models.embeddings import MochiCombinedTimestepCaptionEmbedding, P
|
||||
from diffusers.models.modeling_outputs import Transformer2DModelOutput
|
||||
from diffusers.models.modeling_utils import ModelMixin
|
||||
from diffusers.loaders import PeftAdapterMixin
|
||||
from fastvideo.model.norm import MochiLayerNormContinuous, MochiRMSNormZero, MochiModulatedRMSNorm, MochiRMSNorm
|
||||
from fastvideo.models.mochi_hf.norm import MochiLayerNormContinuous, MochiRMSNormZero, MochiModulatedRMSNorm, MochiRMSNorm
|
||||
from diffusers.models.normalization import AdaLayerNormContinuous
|
||||
|
||||
from fastvideo.utils.parallel_states import get_sequence_parallel_state, nccl_info
|
||||
@@ -21,7 +21,7 @@ from transformers import T5EncoderModel, T5TokenizerFast
|
||||
|
||||
from diffusers.callbacks import MultiPipelineCallbacks, PipelineCallback
|
||||
from diffusers.models.autoencoders import AutoencoderKL
|
||||
from fastvideo.model.modeling_mochi import MochiTransformer3DModel
|
||||
from fastvideo.models.mochi_hf.modeling_mochi import MochiTransformer3DModel
|
||||
|
||||
from diffusers.schedulers import FlowMatchEulerDiscreteScheduler
|
||||
from diffusers.utils import (
|
||||
@@ -200,7 +200,6 @@ class MochiPipeline(DiffusionPipeline, Mochi1LoraLoaderMixin):
|
||||
transformer=transformer,
|
||||
scheduler=scheduler,
|
||||
)
|
||||
# TODO: determine these scaling factors from model parameters
|
||||
self.vae_spatial_scale_factor = 8
|
||||
self.vae_temporal_scale_factor = 6
|
||||
self.patch_size = 2
|
||||
@@ -2,7 +2,7 @@ import json
|
||||
|
||||
import torch.distributed as dist
|
||||
import torch
|
||||
from fastvideo.model.pipeline_mochi import MochiPipeline
|
||||
from fastvideo.models.mochi_hf.pipeline_mochi import MochiPipeline
|
||||
import os
|
||||
from diffusers.utils import export_to_video
|
||||
import argparse
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
import torch
|
||||
from fastvideo.model.pipeline_mochi import MochiPipeline
|
||||
from fastvideo.models.mochi_hf.pipeline_mochi import MochiPipeline
|
||||
import torch.distributed as dist
|
||||
|
||||
from diffusers.utils import export_to_video
|
||||
from fastvideo.utils.parallel_states import initialize_sequence_parallel_state, nccl_info
|
||||
import argparse
|
||||
import os
|
||||
from fastvideo.model.modeling_mochi import MochiTransformer3DModel
|
||||
from fastvideo.models.mochi_hf.modeling_mochi import MochiTransformer3DModel
|
||||
import json
|
||||
from typing import Optional
|
||||
from safetensors.torch import save_file, load_file
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import torch
|
||||
from fastvideo.model.pipeline_mochi import MochiPipeline
|
||||
from fastvideo.model.modeling_mochi import MochiTransformer3DModel
|
||||
from fastvideo.models.mochi_hf.pipeline_mochi import MochiPipeline
|
||||
from fastvideo.models.mochi_hf.modeling_mochi import MochiTransformer3DModel
|
||||
from diffusers.utils import export_to_video, load_image, load_video
|
||||
import argparse
|
||||
from diffusers import FlowMatchEulerDiscreteScheduler
|
||||
|
||||
+5
-9
@@ -8,7 +8,7 @@ from pathlib import Path
|
||||
from fastvideo.utils.parallel_states import initialize_sequence_parallel_state, \
|
||||
destroy_sequence_parallel_group, get_sequence_parallel_state, nccl_info
|
||||
from fastvideo.utils.communications import sp_parallel_dataloader_wrapper, broadcast
|
||||
from fastvideo.model.mochi_latents_utils import normalize_mochi_dit_input
|
||||
from fastvideo.models.mochi_hf.mochi_latents_utils import normalize_mochi_dit_input
|
||||
from fastvideo.utils.validation import log_validation
|
||||
import time
|
||||
from torch.utils.data import DataLoader
|
||||
@@ -30,7 +30,7 @@ from diffusers import (
|
||||
FlowMatchEulerDiscreteScheduler,
|
||||
)
|
||||
from diffusers.optimization import get_scheduler
|
||||
from fastvideo.model.modeling_mochi import MochiTransformer3DModel
|
||||
from fastvideo.models.mochi_hf.modeling_mochi import MochiTransformer3DModel
|
||||
from diffusers.utils import check_min_version
|
||||
from fastvideo.dataset.latent_datasets import LatentDataset, latent_collate_function
|
||||
import torch.distributed as dist
|
||||
@@ -41,7 +41,7 @@ from torch.distributed.fsdp import (
|
||||
)
|
||||
from fastvideo.utils.checkpoint import save_checkpoint, save_lora_checkpoint, resume_lora_optimizer
|
||||
from fastvideo.utils.logging import main_print
|
||||
from fastvideo.model.pipeline_mochi import MochiPipeline
|
||||
from fastvideo.models.mochi_hf.pipeline_mochi import MochiPipeline
|
||||
# Will error if the minimal version of diffusers is not installed. Remove at your own risks.
|
||||
check_min_version("0.31.0")
|
||||
import time
|
||||
@@ -142,10 +142,6 @@ def train_one_step_mochi(transformer, optimizer, lr_scheduler,loader, noise_sche
|
||||
return total_loss, grad_norm.item()
|
||||
|
||||
def main(args):
|
||||
# use LayerNorm, GeLu, SiLu always as fp32 mode
|
||||
# TODO:
|
||||
if args.enable_stable_fp32:
|
||||
raise NotImplementedError("enable_stable_fp32 is not supported now.")
|
||||
torch.backends.cuda.matmul.allow_tf32 = True
|
||||
|
||||
local_rank = int(os.environ['LOCAL_RANK'])
|
||||
@@ -249,7 +245,8 @@ def main(args):
|
||||
transformer, args.resume_from_lora_checkpoint, optimizer
|
||||
)
|
||||
main_print(f"optimizer: {optimizer}")
|
||||
#todo add lr scheduler
|
||||
|
||||
|
||||
lr_scheduler = get_scheduler(
|
||||
args.lr_scheduler,
|
||||
optimizer=optimizer,
|
||||
@@ -383,7 +380,6 @@ if __name__ == "__main__":
|
||||
# text encoder & vae & diffusion model
|
||||
parser.add_argument("--pretrained_model_name_or_path", type=str)
|
||||
parser.add_argument("--cache_dir", type=str, default='./cache_dir')
|
||||
parser.add_argument('--enable_stable_fp32', action='store_true') # TODO
|
||||
|
||||
# diffusion setting
|
||||
parser.add_argument("--ema_decay", type=float, default=0.999)
|
||||
|
||||
@@ -10,7 +10,7 @@ from torch.distributed.checkpoint.default_planner import DefaultSavePlanner, Def
|
||||
from torch.distributed.checkpoint.optimizer import load_sharded_optimizer_state_dict
|
||||
from torch.distributed.fsdp import FullOptimStateDictConfig
|
||||
from peft import LoraConfig, get_peft_model_state_dict, set_peft_model_state_dict
|
||||
from fastvideo.model.pipeline_mochi import MochiPipeline
|
||||
from fastvideo.models.mochi_hf.pipeline_mochi import MochiPipeline
|
||||
def save_checkpoint(model, optimizer, rank, output_dir, step, discriminator=False):
|
||||
|
||||
with FSDP.state_dict_type(
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import argparse
|
||||
import torch
|
||||
from accelerate.logging import get_logger
|
||||
from fastvideo.model.pipeline_mochi import MochiPipeline
|
||||
from fastvideo.models.mochi_hf.pipeline_mochi import MochiPipeline
|
||||
from diffusers.utils import export_to_video
|
||||
import json
|
||||
import os
|
||||
|
||||
@@ -1,328 +0,0 @@
|
||||
import contextlib
|
||||
import copy
|
||||
import random
|
||||
from typing import Any, Dict, Iterable, List, Optional, Union
|
||||
|
||||
from diffusers.utils import (
|
||||
deprecate,
|
||||
is_torchvision_available,
|
||||
is_transformers_available,
|
||||
)
|
||||
|
||||
if is_transformers_available():
|
||||
import transformers
|
||||
|
||||
if is_torchvision_available():
|
||||
from torchvision import transforms
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
|
||||
|
||||
# Adapted from diffusers-style ema https://github.com/huggingface/diffusers/blob/main/src/diffusers/training_utils.py#L263
|
||||
class EMAModel:
|
||||
"""
|
||||
Exponential Moving Average of models weights
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
parameters: Iterable[torch.nn.Parameter],
|
||||
decay: float = 0.9999,
|
||||
min_decay: float = 0.0,
|
||||
update_after_step: int = 0,
|
||||
use_ema_warmup: bool = False,
|
||||
inv_gamma: Union[float, int] = 1.0,
|
||||
power: Union[float, int] = 2 / 3,
|
||||
model_cls: Optional[Any] = None,
|
||||
model_config: Dict[str, Any] = None,
|
||||
**kwargs,
|
||||
):
|
||||
"""
|
||||
Args:
|
||||
parameters (Iterable[torch.nn.Parameter]): The parameters to track.
|
||||
decay (float): The decay factor for the exponential moving average.
|
||||
min_decay (float): The minimum decay factor for the exponential moving average.
|
||||
update_after_step (int): The number of steps to wait before starting to update the EMA weights.
|
||||
use_ema_warmup (bool): Whether to use EMA warmup.
|
||||
inv_gamma (float):
|
||||
Inverse multiplicative factor of EMA warmup. Default: 1. Only used if `use_ema_warmup` is True.
|
||||
power (float): Exponential factor of EMA warmup. Default: 2/3. Only used if `use_ema_warmup` is True.
|
||||
device (Optional[Union[str, torch.device]]): The device to store the EMA weights on. If None, the EMA
|
||||
weights will be stored on CPU.
|
||||
|
||||
@crowsonkb's notes on EMA Warmup:
|
||||
If gamma=1 and power=1, implements a simple average. gamma=1, power=2/3 are good values for models you plan
|
||||
to train for a million or more steps (reaches decay factor 0.999 at 31.6K steps, 0.9999 at 1M steps),
|
||||
gamma=1, power=3/4 for models you plan to train for less (reaches decay factor 0.999 at 10K steps, 0.9999
|
||||
at 215.4k steps).
|
||||
"""
|
||||
|
||||
if isinstance(parameters, torch.nn.Module):
|
||||
deprecation_message = (
|
||||
"Passing a `torch.nn.Module` to `ExponentialMovingAverage` is deprecated. "
|
||||
"Please pass the parameters of the module instead."
|
||||
)
|
||||
deprecate(
|
||||
"passing a `torch.nn.Module` to `ExponentialMovingAverage`",
|
||||
"1.0.0",
|
||||
deprecation_message,
|
||||
standard_warn=False,
|
||||
)
|
||||
parameters = parameters.parameters()
|
||||
|
||||
# set use_ema_warmup to True if a torch.nn.Module is passed for backwards compatibility
|
||||
use_ema_warmup = True
|
||||
|
||||
if kwargs.get("max_value", None) is not None:
|
||||
deprecation_message = "The `max_value` argument is deprecated. Please use `decay` instead."
|
||||
deprecate("max_value", "1.0.0", deprecation_message, standard_warn=False)
|
||||
decay = kwargs["max_value"]
|
||||
|
||||
if kwargs.get("min_value", None) is not None:
|
||||
deprecation_message = "The `min_value` argument is deprecated. Please use `min_decay` instead."
|
||||
deprecate("min_value", "1.0.0", deprecation_message, standard_warn=False)
|
||||
min_decay = kwargs["min_value"]
|
||||
|
||||
parameters = list(parameters)
|
||||
self.shadow_params = [p.clone().detach() for p in parameters]
|
||||
|
||||
if kwargs.get("device", None) is not None:
|
||||
deprecation_message = "The `device` argument is deprecated. Please use `to` instead."
|
||||
deprecate("device", "1.0.0", deprecation_message, standard_warn=False)
|
||||
self.to(device=kwargs["device"])
|
||||
|
||||
self.temp_stored_params = None
|
||||
|
||||
self.decay = decay
|
||||
self.min_decay = min_decay
|
||||
self.update_after_step = update_after_step
|
||||
self.use_ema_warmup = use_ema_warmup
|
||||
self.inv_gamma = inv_gamma
|
||||
self.power = power
|
||||
self.optimization_step = 0
|
||||
self.cur_decay_value = None # set in `step()`
|
||||
|
||||
self.model_cls = model_cls
|
||||
self.model_config = model_config
|
||||
|
||||
@classmethod
|
||||
def extract_ema_kwargs(cls, kwargs):
|
||||
"""
|
||||
Extracts the EMA kwargs from the kwargs of a class method.
|
||||
"""
|
||||
ema_kwargs = {}
|
||||
for key in [
|
||||
"decay",
|
||||
"min_decay",
|
||||
"optimization_step",
|
||||
"update_after_step",
|
||||
"use_ema_warmup",
|
||||
"inv_gamma",
|
||||
"power",
|
||||
]:
|
||||
if kwargs.get(key, None) is not None:
|
||||
ema_kwargs[key] = kwargs.pop(key)
|
||||
return ema_kwargs
|
||||
|
||||
@classmethod
|
||||
def from_pretrained(cls, path, model_cls) -> "EMAModel":
|
||||
config = model_cls.load_config(path)
|
||||
ema_kwargs = cls.extract_ema_kwargs(config)
|
||||
model = model_cls.from_pretrained(path)
|
||||
|
||||
ema_model = cls(model.parameters(), model_cls=model_cls, model_config=config)
|
||||
|
||||
ema_model.load_state_dict(ema_kwargs)
|
||||
return ema_model
|
||||
|
||||
def save_pretrained(self, path):
|
||||
if self.model_cls is None:
|
||||
raise ValueError("`save_pretrained` can only be used if `model_cls` was defined at __init__.")
|
||||
|
||||
if self.model_config is None:
|
||||
raise ValueError("`save_pretrained` can only be used if `model_config` was defined at __init__.")
|
||||
|
||||
model = self.model_cls.from_config(self.model_config)
|
||||
state_dict = self.state_dict()
|
||||
state_dict.pop("shadow_params", None)
|
||||
|
||||
model.register_to_config(**state_dict)
|
||||
self.copy_to(model.parameters())
|
||||
model.save_pretrained(path)
|
||||
|
||||
def get_decay(self, optimization_step: int) -> float:
|
||||
"""
|
||||
Compute the decay factor for the exponential moving average.
|
||||
"""
|
||||
step = max(0, optimization_step - self.update_after_step - 1)
|
||||
|
||||
if step <= 0:
|
||||
return 0.0
|
||||
|
||||
if self.use_ema_warmup:
|
||||
cur_decay_value = 1 - (1 + step / self.inv_gamma) ** -self.power
|
||||
else:
|
||||
cur_decay_value = (1 + step) / (10 + step)
|
||||
|
||||
cur_decay_value = min(cur_decay_value, self.decay)
|
||||
# make sure decay is not smaller than min_decay
|
||||
cur_decay_value = max(cur_decay_value, self.min_decay)
|
||||
return cur_decay_value
|
||||
|
||||
@torch.no_grad()
|
||||
def step(self, parameters: Iterable[torch.nn.Parameter]):
|
||||
if isinstance(parameters, torch.nn.Module):
|
||||
deprecation_message = (
|
||||
"Passing a `torch.nn.Module` to `ExponentialMovingAverage.step` is deprecated. "
|
||||
"Please pass the parameters of the module instead."
|
||||
)
|
||||
deprecate(
|
||||
"passing a `torch.nn.Module` to `ExponentialMovingAverage.step`",
|
||||
"1.0.0",
|
||||
deprecation_message,
|
||||
standard_warn=False,
|
||||
)
|
||||
parameters = parameters.parameters()
|
||||
|
||||
parameters = list(parameters)
|
||||
|
||||
self.optimization_step += 1
|
||||
|
||||
# Compute the decay factor for the exponential moving average.
|
||||
decay = self.get_decay(self.optimization_step)
|
||||
self.cur_decay_value = decay
|
||||
one_minus_decay = 1 - decay
|
||||
|
||||
context_manager = contextlib.nullcontext
|
||||
if is_transformers_available() and transformers.integrations.is_deepspeed_zero3_enabled():
|
||||
import deepspeed
|
||||
|
||||
for s_param, param in zip(self.shadow_params, parameters):
|
||||
if is_transformers_available() and transformers.integrations.is_deepspeed_zero3_enabled():
|
||||
context_manager = deepspeed.zero.GatheredParameters(param, modifier_rank=None)
|
||||
|
||||
with context_manager():
|
||||
if param.requires_grad:
|
||||
s_param.sub_(one_minus_decay * (s_param - param))
|
||||
else:
|
||||
s_param.copy_(param)
|
||||
|
||||
def copy_to(self, parameters: Iterable[torch.nn.Parameter]) -> None:
|
||||
"""
|
||||
Copy current averaged parameters into given collection of parameters.
|
||||
|
||||
Args:
|
||||
parameters: Iterable of `torch.nn.Parameter`; the parameters to be
|
||||
updated with the stored moving averages. If `None`, the parameters with which this
|
||||
`ExponentialMovingAverage` was initialized will be used.
|
||||
"""
|
||||
parameters = list(parameters)
|
||||
for s_param, param in zip(self.shadow_params, parameters):
|
||||
param.data.copy_(s_param.to(param.device).data)
|
||||
|
||||
|
||||
def to(self, device=None, dtype=None) -> None:
|
||||
r"""Move internal buffers of the ExponentialMovingAverage to `device`.
|
||||
|
||||
Args:
|
||||
device: like `device` argument to `torch.Tensor.to`
|
||||
"""
|
||||
# .to() on the tensors handles None correctly
|
||||
self.shadow_params = [
|
||||
p.to(device=device, dtype=dtype) if p.is_floating_point() else p.to(device=device)
|
||||
for p in self.shadow_params
|
||||
]
|
||||
|
||||
def state_dict(self) -> dict:
|
||||
r"""
|
||||
Returns the state of the ExponentialMovingAverage as a dict. This method is used by accelerate during
|
||||
checkpointing to save the ema state dict.
|
||||
"""
|
||||
# Following PyTorch conventions, references to tensors are returned:
|
||||
# "returns a reference to the state and not its copy!" -
|
||||
# https://pytorch.org/tutorials/beginner/saving_loading_models.html#what-is-a-state-dict
|
||||
return {
|
||||
"decay": self.decay,
|
||||
"min_decay": self.min_decay,
|
||||
"optimization_step": self.optimization_step,
|
||||
"update_after_step": self.update_after_step,
|
||||
"use_ema_warmup": self.use_ema_warmup,
|
||||
"inv_gamma": self.inv_gamma,
|
||||
"power": self.power,
|
||||
"shadow_params": self.shadow_params,
|
||||
}
|
||||
|
||||
def store(self, parameters: Iterable[torch.nn.Parameter]) -> None:
|
||||
r"""
|
||||
Args:
|
||||
Save the current parameters for restoring later.
|
||||
parameters: Iterable of `torch.nn.Parameter`; the parameters to be
|
||||
temporarily stored.
|
||||
"""
|
||||
self.temp_stored_params = [param.detach().cpu().clone() for param in parameters]
|
||||
|
||||
def restore(self, parameters: Iterable[torch.nn.Parameter]) -> None:
|
||||
r"""
|
||||
Args:
|
||||
Restore the parameters stored with the `store` method. Useful to validate the model with EMA parameters without:
|
||||
affecting the original optimization process. Store the parameters before the `copy_to()` method. After
|
||||
validation (or model saving), use this to restore the former parameters.
|
||||
parameters: Iterable of `torch.nn.Parameter`; the parameters to be
|
||||
updated with the stored parameters. If `None`, the parameters with which this
|
||||
`ExponentialMovingAverage` was initialized will be used.
|
||||
"""
|
||||
if self.temp_stored_params is None:
|
||||
raise RuntimeError("This ExponentialMovingAverage has no `store()`ed weights " "to `restore()`")
|
||||
for c_param, param in zip(self.temp_stored_params, parameters):
|
||||
param.data.copy_(c_param.data)
|
||||
|
||||
# Better memory-wise.
|
||||
self.temp_stored_params = None
|
||||
|
||||
def load_state_dict(self, state_dict: dict) -> None:
|
||||
r"""
|
||||
Args:
|
||||
Loads the ExponentialMovingAverage state. This method is used by accelerate during checkpointing to save the
|
||||
ema state dict.
|
||||
state_dict (dict): EMA state. Should be an object returned
|
||||
from a call to :meth:`state_dict`.
|
||||
"""
|
||||
# deepcopy, to be consistent with module API
|
||||
state_dict = copy.deepcopy(state_dict)
|
||||
|
||||
self.decay = state_dict.get("decay", self.decay)
|
||||
if self.decay < 0.0 or self.decay > 1.0:
|
||||
raise ValueError("Decay must be between 0 and 1")
|
||||
|
||||
self.min_decay = state_dict.get("min_decay", self.min_decay)
|
||||
if not isinstance(self.min_decay, float):
|
||||
raise ValueError("Invalid min_decay")
|
||||
|
||||
self.optimization_step = state_dict.get("optimization_step", self.optimization_step)
|
||||
if not isinstance(self.optimization_step, int):
|
||||
raise ValueError("Invalid optimization_step")
|
||||
|
||||
self.update_after_step = state_dict.get("update_after_step", self.update_after_step)
|
||||
if not isinstance(self.update_after_step, int):
|
||||
raise ValueError("Invalid update_after_step")
|
||||
|
||||
self.use_ema_warmup = state_dict.get("use_ema_warmup", self.use_ema_warmup)
|
||||
if not isinstance(self.use_ema_warmup, bool):
|
||||
raise ValueError("Invalid use_ema_warmup")
|
||||
|
||||
self.inv_gamma = state_dict.get("inv_gamma", self.inv_gamma)
|
||||
if not isinstance(self.inv_gamma, (float, int)):
|
||||
raise ValueError("Invalid inv_gamma")
|
||||
|
||||
self.power = state_dict.get("power", self.power)
|
||||
if not isinstance(self.power, (float, int)):
|
||||
raise ValueError("Invalid power")
|
||||
|
||||
shadow_params = state_dict.get("shadow_params", None)
|
||||
if shadow_params is not None:
|
||||
self.shadow_params = shadow_params
|
||||
if not isinstance(self.shadow_params, list):
|
||||
raise ValueError("shadow_params must be a list")
|
||||
if not all(isinstance(p, torch.Tensor) for p in self.shadow_params):
|
||||
raise ValueError("shadow_params must all be Tensors")
|
||||
@@ -1,471 +0,0 @@
|
||||
import os
|
||||
|
||||
import torch
|
||||
|
||||
import os
|
||||
import math
|
||||
import torch
|
||||
import logging
|
||||
import random
|
||||
import subprocess
|
||||
import numpy as np
|
||||
import torch.distributed as dist
|
||||
|
||||
# from torch._six import inf
|
||||
from torch import inf
|
||||
from PIL import Image
|
||||
from typing import Union, Iterable
|
||||
import collections
|
||||
from collections import OrderedDict
|
||||
from torch.utils.tensorboard import SummaryWriter
|
||||
|
||||
from diffusers.utils import is_bs4_available, is_ftfy_available
|
||||
|
||||
import html
|
||||
import re
|
||||
import urllib.parse as ul
|
||||
|
||||
if is_bs4_available():
|
||||
from bs4 import BeautifulSoup
|
||||
|
||||
if is_ftfy_available():
|
||||
import ftfy
|
||||
|
||||
_tensor_or_tensors = Union[torch.Tensor, Iterable[torch.Tensor]]
|
||||
|
||||
def to_2tuple(x):
|
||||
if isinstance(x, collections.abc.Iterable):
|
||||
return x
|
||||
return (x, x)
|
||||
|
||||
def find_model(model_name):
|
||||
"""
|
||||
Finds a pre-trained Latte model, downloading it if necessary. Alternatively, loads a model from a local path.
|
||||
"""
|
||||
assert os.path.isfile(model_name), f'Could not find Latte checkpoint at {model_name}'
|
||||
checkpoint = torch.load(model_name, map_location=lambda storage, loc: storage)
|
||||
|
||||
# if "ema" in checkpoint: # supports checkpoints from train.py
|
||||
# print('Using Ema!')
|
||||
# checkpoint = checkpoint["ema"]
|
||||
# else:
|
||||
print('Using model!')
|
||||
checkpoint = checkpoint['model']
|
||||
return checkpoint
|
||||
|
||||
#################################################################################
|
||||
# Training Clip Gradients #
|
||||
#################################################################################
|
||||
|
||||
def get_grad_norm(
|
||||
parameters: _tensor_or_tensors, norm_type: float = 2.0) -> torch.Tensor:
|
||||
r"""
|
||||
Copy from torch.nn.utils.clip_grad_norm_
|
||||
|
||||
Clips gradient norm of an iterable of parameters.
|
||||
|
||||
The norm is computed over all gradients together, as if they were
|
||||
concatenated into a single vector. Gradients are modified in-place.
|
||||
|
||||
Args:
|
||||
parameters (Iterable[Tensor] or Tensor): an iterable of Tensors or a
|
||||
single Tensor that will have gradients normalized
|
||||
max_norm (float or int): max norm of the gradients
|
||||
norm_type (float or int): type of the used p-norm. Can be ``'inf'`` for
|
||||
infinity norm.
|
||||
error_if_nonfinite (bool): if True, an error is thrown if the total
|
||||
norm of the gradients from :attr:`parameters` is ``nan``,
|
||||
``inf``, or ``-inf``. Default: False (will switch to True in the future)
|
||||
|
||||
Returns:
|
||||
Total norm of the parameter gradients (viewed as a single vector).
|
||||
"""
|
||||
if isinstance(parameters, torch.Tensor):
|
||||
parameters = [parameters]
|
||||
grads = [p.grad for p in parameters if p.grad is not None]
|
||||
norm_type = float(norm_type)
|
||||
if len(grads) == 0:
|
||||
return torch.tensor(0.)
|
||||
device = grads[0].device
|
||||
if norm_type == inf:
|
||||
norms = [g.detach().abs().max().to(device) for g in grads]
|
||||
total_norm = norms[0] if len(norms) == 1 else torch.max(torch.stack(norms))
|
||||
else:
|
||||
total_norm = torch.norm(torch.stack([torch.norm(g.detach(), norm_type).to(device) for g in grads]), norm_type)
|
||||
return total_norm
|
||||
|
||||
|
||||
def clip_grad_norm_(
|
||||
parameters: _tensor_or_tensors, max_norm: float, norm_type: float = 2.0,
|
||||
error_if_nonfinite: bool = False, clip_grad=True) -> torch.Tensor:
|
||||
r"""
|
||||
Copy from torch.nn.utils.clip_grad_norm_
|
||||
|
||||
Clips gradient norm of an iterable of parameters.
|
||||
|
||||
The norm is computed over all gradients together, as if they were
|
||||
concatenated into a single vector. Gradients are modified in-place.
|
||||
|
||||
Args:
|
||||
parameters (Iterable[Tensor] or Tensor): an iterable of Tensors or a
|
||||
single Tensor that will have gradients normalized
|
||||
max_norm (float or int): max norm of the gradients
|
||||
norm_type (float or int): type of the used p-norm. Can be ``'inf'`` for
|
||||
infinity norm.
|
||||
error_if_nonfinite (bool): if True, an error is thrown if the total
|
||||
norm of the gradients from :attr:`parameters` is ``nan``,
|
||||
``inf``, or ``-inf``. Default: False (will switch to True in the future)
|
||||
|
||||
Returns:
|
||||
Total norm of the parameter gradients (viewed as a single vector).
|
||||
"""
|
||||
if isinstance(parameters, torch.Tensor):
|
||||
parameters = [parameters]
|
||||
grads = [p.grad for p in parameters if p.grad is not None]
|
||||
max_norm = float(max_norm)
|
||||
norm_type = float(norm_type)
|
||||
if len(grads) == 0:
|
||||
return torch.tensor(0.)
|
||||
device = grads[0].device
|
||||
if norm_type == inf:
|
||||
norms = [g.detach().abs().max().to(device) for g in grads]
|
||||
total_norm = norms[0] if len(norms) == 1 else torch.max(torch.stack(norms))
|
||||
else:
|
||||
total_norm = torch.norm(torch.stack([torch.norm(g.detach(), norm_type).to(device) for g in grads]), norm_type)
|
||||
|
||||
if clip_grad:
|
||||
if error_if_nonfinite and torch.logical_or(total_norm.isnan(), total_norm.isinf()):
|
||||
raise RuntimeError(
|
||||
f'The total norm of order {norm_type} for gradients from '
|
||||
'`parameters` is non-finite, so it cannot be clipped. To disable '
|
||||
'this error and scale the gradients by the non-finite norm anyway, '
|
||||
'set `error_if_nonfinite=False`')
|
||||
clip_coef = max_norm / (total_norm + 1e-6)
|
||||
# Note: multiplying by the clamped coef is redundant when the coef is clamped to 1, but doing so
|
||||
# avoids a `if clip_coef < 1:` conditional which can require a CPU <=> device synchronization
|
||||
# when the gradients do not reside in CPU memory.
|
||||
clip_coef_clamped = torch.clamp(clip_coef, max=1.0)
|
||||
for g in grads:
|
||||
g.detach().mul_(clip_coef_clamped.to(g.device))
|
||||
# gradient_cliped = torch.norm(torch.stack([torch.norm(g.detach(), norm_type).to(device) for g in grads]), norm_type)
|
||||
# print(gradient_cliped)
|
||||
return total_norm
|
||||
|
||||
|
||||
def get_experiment_dir(root_dir, args):
|
||||
# if args.pretrained is not None and 'Latte-XL-2-256x256.pt' not in args.pretrained:
|
||||
# root_dir += '-WOPRE'
|
||||
if args.use_compile:
|
||||
root_dir += '-Compile' # speedup by torch compile
|
||||
if args.attention_mode:
|
||||
root_dir += f'-{args.attention_mode.upper()}'
|
||||
# if args.enable_xformers_memory_efficient_attention:
|
||||
# root_dir += '-Xfor'
|
||||
if args.gradient_checkpointing:
|
||||
root_dir += '-Gc'
|
||||
if args.mixed_precision:
|
||||
root_dir += f'-{args.mixed_precision.upper()}'
|
||||
root_dir += f'-{args.max_image_size}'
|
||||
return root_dir
|
||||
|
||||
def get_precision(args):
|
||||
if args.mixed_precision == "bf16":
|
||||
dtype = torch.bfloat16
|
||||
elif args.mixed_precision == "fp16":
|
||||
dtype = torch.float16
|
||||
else:
|
||||
dtype = torch.float32
|
||||
return dtype
|
||||
|
||||
#################################################################################
|
||||
# Training Logger #
|
||||
#################################################################################
|
||||
|
||||
def create_logger(logging_dir):
|
||||
"""
|
||||
Create a logger that writes to a log file and stdout.
|
||||
"""
|
||||
if dist.get_rank() == 0: # real logger
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
# format='[\033[34m%(asctime)s\033[0m] %(message)s',
|
||||
format='[%(asctime)s] %(message)s',
|
||||
datefmt='%Y-%m-%d %H:%M:%S',
|
||||
handlers=[logging.StreamHandler(), logging.FileHandler(f"{logging_dir}/log.txt")]
|
||||
)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
else: # dummy logger (does nothing)
|
||||
logger = logging.getLogger(__name__)
|
||||
logger.addHandler(logging.NullHandler())
|
||||
return logger
|
||||
|
||||
|
||||
def create_tensorboard(tensorboard_dir):
|
||||
"""
|
||||
Create a tensorboard that saves losses.
|
||||
"""
|
||||
if dist.get_rank() == 0: # real tensorboard
|
||||
# tensorboard
|
||||
writer = SummaryWriter(tensorboard_dir)
|
||||
|
||||
return writer
|
||||
|
||||
|
||||
def write_tensorboard(writer, *args):
|
||||
'''
|
||||
write the loss information to a tensorboard file.
|
||||
Only for pytorch DDP mode.
|
||||
'''
|
||||
if dist.get_rank() == 0: # real tensorboard
|
||||
writer.add_scalar(args[0], args[1], args[2])
|
||||
|
||||
|
||||
#################################################################################
|
||||
# EMA Update/ DDP Training Utils #
|
||||
#################################################################################
|
||||
|
||||
@torch.no_grad()
|
||||
def update_ema(ema_model, model, decay=0.9999):
|
||||
"""
|
||||
Step the EMA model towards the current model.
|
||||
"""
|
||||
ema_params = OrderedDict(ema_model.named_parameters())
|
||||
model_params = OrderedDict(model.named_parameters())
|
||||
|
||||
for name, param in model_params.items():
|
||||
# TODO: Consider applying only to params that require_grad to avoid small numerical changes of pos_embed
|
||||
ema_params[name].mul_(decay).add_(param.data, alpha=1 - decay)
|
||||
|
||||
|
||||
def requires_grad(model, flag=True):
|
||||
"""
|
||||
Set requires_grad flag for all parameters in a model.
|
||||
"""
|
||||
for p in model.parameters():
|
||||
p.requires_grad = flag
|
||||
|
||||
|
||||
def cleanup():
|
||||
"""
|
||||
End DDP training.
|
||||
"""
|
||||
dist.destroy_process_group()
|
||||
|
||||
|
||||
def setup_distributed(backend="nccl", port=None):
|
||||
"""Initialize distributed training environment.
|
||||
support both slurm and torch.distributed.launch
|
||||
see torch.distributed.init_process_group() for more details
|
||||
"""
|
||||
num_gpus = torch.cuda.device_count()
|
||||
|
||||
if "SLURM_JOB_ID" in os.environ:
|
||||
rank = int(os.environ["SLURM_PROCID"])
|
||||
world_size = int(os.environ["SLURM_NTASKS"])
|
||||
node_list = os.environ["SLURM_NODELIST"]
|
||||
addr = subprocess.getoutput(f"scontrol show hostname {node_list} | head -n1")
|
||||
# specify master port
|
||||
if port is not None:
|
||||
os.environ["MASTER_PORT"] = str(port)
|
||||
elif "MASTER_PORT" not in os.environ:
|
||||
# os.environ["MASTER_PORT"] = "29566"
|
||||
os.environ["MASTER_PORT"] = str(29567 + num_gpus)
|
||||
if "MASTER_ADDR" not in os.environ:
|
||||
os.environ["MASTER_ADDR"] = addr
|
||||
os.environ["WORLD_SIZE"] = str(world_size)
|
||||
os.environ["LOCAL_RANK"] = str(rank % num_gpus)
|
||||
os.environ["RANK"] = str(rank)
|
||||
else:
|
||||
rank = int(os.environ["RANK"])
|
||||
world_size = int(os.environ["WORLD_SIZE"])
|
||||
|
||||
# torch.cuda.set_device(rank % num_gpus)
|
||||
|
||||
dist.init_process_group(
|
||||
backend=backend,
|
||||
world_size=world_size,
|
||||
rank=rank,
|
||||
)
|
||||
|
||||
|
||||
#################################################################################
|
||||
# Testing Utils #
|
||||
#################################################################################
|
||||
|
||||
def save_video_grid(video, nrow=None):
|
||||
b, t, h, w, c = video.shape
|
||||
|
||||
if nrow is None:
|
||||
nrow = math.ceil(math.sqrt(b))
|
||||
ncol = math.ceil(b / nrow)
|
||||
padding = 1
|
||||
video_grid = torch.zeros((t, (padding + h) * nrow + padding,
|
||||
(padding + w) * ncol + padding, c), dtype=torch.uint8)
|
||||
|
||||
print(video_grid.shape)
|
||||
for i in range(b):
|
||||
r = i // ncol
|
||||
c = i % ncol
|
||||
start_r = (padding + h) * r
|
||||
start_c = (padding + w) * c
|
||||
video_grid[:, start_r:start_r + h, start_c:start_c + w] = video[i]
|
||||
|
||||
return video_grid
|
||||
|
||||
|
||||
#################################################################################
|
||||
# MMCV Utils #
|
||||
#################################################################################
|
||||
|
||||
|
||||
def collect_env():
|
||||
# Copyright (c) OpenMMLab. All rights reserved.
|
||||
from mmcv.utils import collect_env as collect_base_env
|
||||
from mmcv.utils import get_git_hash
|
||||
"""Collect the information of the running environments."""
|
||||
|
||||
env_info = collect_base_env()
|
||||
env_info['MMClassification'] = get_git_hash()[:7]
|
||||
|
||||
for name, val in env_info.items():
|
||||
print(f'{name}: {val}')
|
||||
|
||||
print(torch.cuda.get_arch_list())
|
||||
print(torch.version.cuda)
|
||||
|
||||
|
||||
#################################################################################
|
||||
# Pixart-alpha Utils #
|
||||
#################################################################################
|
||||
|
||||
|
||||
bad_punct_regex = re.compile(r'['+'#®•©™&@·º½¾¿¡§~'+'\)'+'\('+'\]'+'\['+'\}'+'\{'+'\|'+'\\'+'\/'+'\*' + r']{1,}') # noqa
|
||||
|
||||
def text_preprocessing(text, support_Chinese=True):
|
||||
# The exact text cleaning as was in the training stage:
|
||||
text = clean_caption(text, support_Chinese=support_Chinese)
|
||||
text = clean_caption(text, support_Chinese=support_Chinese)
|
||||
return text
|
||||
|
||||
def basic_clean(text):
|
||||
text = ftfy.fix_text(text)
|
||||
text = html.unescape(html.unescape(text))
|
||||
return text.strip()
|
||||
|
||||
def clean_caption(caption, support_Chinese=True):
|
||||
caption = str(caption)
|
||||
caption = ul.unquote_plus(caption)
|
||||
caption = caption.strip().lower()
|
||||
caption = re.sub('<person>', 'person', caption)
|
||||
# urls:
|
||||
caption = re.sub(
|
||||
r'\b((?:https?:(?:\/{1,3}|[a-zA-Z0-9%])|[a-zA-Z0-9.\-]+[.](?:com|co|ru|net|org|edu|gov|it)[\w/-]*\b\/?(?!@)))', # noqa
|
||||
'', caption) # regex for urls
|
||||
caption = re.sub(
|
||||
r'\b((?:www:(?:\/{1,3}|[a-zA-Z0-9%])|[a-zA-Z0-9.\-]+[.](?:com|co|ru|net|org|edu|gov|it)[\w/-]*\b\/?(?!@)))', # noqa
|
||||
'', caption) # regex for urls
|
||||
# html:
|
||||
caption = BeautifulSoup(caption, features='html.parser').text
|
||||
|
||||
# @<nickname>
|
||||
caption = re.sub(r'@[\w\d]+\b', '', caption)
|
||||
|
||||
# 31C0—31EF CJK Strokes
|
||||
# 31F0—31FF Katakana Phonetic Extensions
|
||||
# 3200—32FF Enclosed CJK Letters and Months
|
||||
# 3300—33FF CJK Compatibility
|
||||
# 3400—4DBF CJK Unified Ideographs Extension A
|
||||
# 4DC0—4DFF Yijing Hexagram Symbols
|
||||
# 4E00—9FFF CJK Unified Ideographs
|
||||
caption = re.sub(r'[\u31c0-\u31ef]+', '', caption)
|
||||
caption = re.sub(r'[\u31f0-\u31ff]+', '', caption)
|
||||
caption = re.sub(r'[\u3200-\u32ff]+', '', caption)
|
||||
caption = re.sub(r'[\u3300-\u33ff]+', '', caption)
|
||||
caption = re.sub(r'[\u3400-\u4dbf]+', '', caption)
|
||||
caption = re.sub(r'[\u4dc0-\u4dff]+', '', caption)
|
||||
if not support_Chinese:
|
||||
caption = re.sub(r'[\u4e00-\u9fff]+', '', caption) # Chinese
|
||||
#######################################################
|
||||
|
||||
# все виды тире / all types of dash --> "-"
|
||||
caption = re.sub(
|
||||
r'[\u002D\u058A\u05BE\u1400\u1806\u2010-\u2015\u2E17\u2E1A\u2E3A\u2E3B\u2E40\u301C\u3030\u30A0\uFE31\uFE32\uFE58\uFE63\uFF0D]+', # noqa
|
||||
'-', caption)
|
||||
|
||||
# кавычки к одному стандарту
|
||||
caption = re.sub(r'[`´«»“”¨]', '"', caption)
|
||||
caption = re.sub(r'[‘’]', "'", caption)
|
||||
|
||||
# "
|
||||
caption = re.sub(r'"?', '', caption)
|
||||
# &
|
||||
caption = re.sub(r'&', '', caption)
|
||||
|
||||
# ip adresses:
|
||||
caption = re.sub(r'\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}', ' ', caption)
|
||||
|
||||
# article ids:
|
||||
caption = re.sub(r'\d:\d\d\s+$', '', caption)
|
||||
|
||||
# \n
|
||||
caption = re.sub(r'\\n', ' ', caption)
|
||||
|
||||
# "#123"
|
||||
caption = re.sub(r'#\d{1,3}\b', '', caption)
|
||||
# "#12345.."
|
||||
caption = re.sub(r'#\d{5,}\b', '', caption)
|
||||
# "123456.."
|
||||
caption = re.sub(r'\b\d{6,}\b', '', caption)
|
||||
# filenames:
|
||||
caption = re.sub(r'[\S]+\.(?:png|jpg|jpeg|bmp|webp|eps|pdf|apk|mp4)', '', caption)
|
||||
|
||||
#
|
||||
caption = re.sub(r'[\"\']{2,}', r'"', caption) # """AUSVERKAUFT"""
|
||||
caption = re.sub(r'[\.]{2,}', r' ', caption) # """AUSVERKAUFT"""
|
||||
|
||||
caption = re.sub(bad_punct_regex, r' ', caption) # ***AUSVERKAUFT***, #AUSVERKAUFT
|
||||
caption = re.sub(r'\s+\.\s+', r' ', caption) # " . "
|
||||
|
||||
# this-is-my-cute-cat / this_is_my_cute_cat
|
||||
regex2 = re.compile(r'(?:\-|\_)')
|
||||
if len(re.findall(regex2, caption)) > 3:
|
||||
caption = re.sub(regex2, ' ', caption)
|
||||
|
||||
caption = basic_clean(caption)
|
||||
|
||||
caption = re.sub(r'\b[a-zA-Z]{1,3}\d{3,15}\b', '', caption) # jc6640
|
||||
caption = re.sub(r'\b[a-zA-Z]+\d+[a-zA-Z]+\b', '', caption) # jc6640vc
|
||||
caption = re.sub(r'\b\d+[a-zA-Z]+\d+\b', '', caption) # 6640vc231
|
||||
|
||||
caption = re.sub(r'(worldwide\s+)?(free\s+)?shipping', '', caption)
|
||||
caption = re.sub(r'(free\s)?download(\sfree)?', '', caption)
|
||||
caption = re.sub(r'\bclick\b\s(?:for|on)\s\w+', '', caption)
|
||||
caption = re.sub(r'\b(?:png|jpg|jpeg|bmp|webp|eps|pdf|apk|mp4)(\simage[s]?)?', '', caption)
|
||||
caption = re.sub(r'\bpage\s+\d+\b', '', caption)
|
||||
|
||||
caption = re.sub(r'\b\d*[a-zA-Z]+\d+[a-zA-Z]+\d+[a-zA-Z\d]*\b', r' ', caption) # j2d1a2a...
|
||||
|
||||
caption = re.sub(r'\b\d+\.?\d*[xх×]\d+\.?\d*\b', '', caption)
|
||||
|
||||
caption = re.sub(r'\b\s+\:\s+', r': ', caption)
|
||||
caption = re.sub(r'(\D[,\./])\b', r'\1 ', caption)
|
||||
caption = re.sub(r'\s+', ' ', caption)
|
||||
|
||||
caption.strip()
|
||||
|
||||
caption = re.sub(r'^[\"\']([\w\W]+)[\"\']$', r'\1', caption)
|
||||
caption = re.sub(r'^[\'\_,\-\:;]', r'', caption)
|
||||
caption = re.sub(r'[\'\_,\-\:\-\+]$', r'', caption)
|
||||
caption = re.sub(r'^\.\S+$', '', caption)
|
||||
|
||||
return caption.strip()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
|
||||
# caption = re.sub(r'[\u4e00-\u9fff]+', '', caption)
|
||||
a = "امرأة مسنة بشعر أبيض ووجه مليء بالتجاعيد تجلس داخل سيارة قديمة الطراز، تنظر من خلال النافذة الجانبية بتعبير تأملي أو حزين قليلاً."
|
||||
print(a)
|
||||
print(text_preprocessing(a))
|
||||
|
||||
@@ -7,7 +7,7 @@ from einops import rearrange
|
||||
from fastvideo.utils.parallel_states import get_sequence_parallel_state, nccl_info
|
||||
from fastvideo.utils.communications import all_gather
|
||||
from diffusers.utils.torch_utils import randn_tensor
|
||||
from fastvideo.model.pipeline_mochi import linear_quadratic_schedule, retrieve_timesteps
|
||||
from fastvideo.models.mochi_hf.pipeline_mochi import linear_quadratic_schedule, retrieve_timesteps
|
||||
from tqdm import tqdm
|
||||
from diffusers.video_processor import VideoProcessor
|
||||
from diffusers import (
|
||||
|
||||
@@ -1,24 +1,5 @@
|
||||
|
||||
|
||||
num_gpus=4
|
||||
|
||||
torchrun --nnodes=1 --nproc_per_node=$num_gpus --master_port 29503 \
|
||||
fastvideo/sample/sample_t2v_mochi.py \
|
||||
--model_path data/mochi \
|
||||
--prompt_path data/prompt.txt \
|
||||
--transformer_path data/outputs/video_distill_synthetic/checkpoint-1500 \
|
||||
--num_frames 163 \
|
||||
--height 480 \
|
||||
--width 848 \
|
||||
--num_inference_steps 8 \
|
||||
--guidance_scale 4.5 \
|
||||
--output_path outputs_video/distill_lq_163_1500_precision_stochastic_0.7 \
|
||||
--shift 8 \
|
||||
--seed 12345 \
|
||||
--scheduler_type "pcm_linear_quadratic"
|
||||
|
||||
|
||||
|
||||
|
||||
num_gpus=4
|
||||
|
||||
|
||||
@@ -1,11 +0,0 @@
|
||||
import json
|
||||
import os
|
||||
|
||||
path = "data/outputs/BW_Testrun/checkpoint-0/config.json"
|
||||
|
||||
with open(path, 'r') as f:
|
||||
data = json.load(f)
|
||||
|
||||
# save with indent
|
||||
with open(path, 'w') as f:
|
||||
json.dump(data, f, indent=4)
|
||||
+1
-5
@@ -21,12 +21,8 @@ dependencies = [
|
||||
"timm==1.0.11", "torchdiffeq==0.2.4", "torchmetrics==1.5.1", "tqdm==4.66.5", "urllib3==2.2.0", "uvicorn==0.32.0",
|
||||
"scikit-video==1.1.11", "imageio-ffmpeg==0.5.1", "sentencepiece==0.2.0", "beautifulsoup4==4.12.3", "ftfy==6.3.0",
|
||||
"moviepy==1.0.3", "wandb==0.18.5", "tensorboard==2.18.0", "pydantic==2.9.2", "gradio==5.3.0", "huggingface_hub==0.26.1", "protobuf==5.28.3",
|
||||
"watch", "gpustat", "peft==0.13.2", "liger_kernel==0.4.1", "einops==0.8.0", "wheel==0.44.0"
|
||||
]
|
||||
"watch", "gpustat", "peft==0.13.2", "liger_kernel==0.4.1", "einops==0.8.0", "wheel==0.44.0"]
|
||||
|
||||
[project.optional-dependencies]
|
||||
train = ["deepspeed==0.15.3"]
|
||||
dev = ["mypy==1.8.0"]
|
||||
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
|
||||
@@ -1,56 +0,0 @@
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export NCCL_DEBUG=INFO
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=[MASTER_NODE_IP_ADDRESS]:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path data/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "data/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "data/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--uncond_prompt_dir "data/validation_embeddings/uncond_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 250\
|
||||
--validation_sampling_steps 8 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--ema_decay 0.999\
|
||||
--log_validation\
|
||||
--output_dir="data/outputs/lq_euler_50_thresh_0.025"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale 4.5 \
|
||||
--num_euler_timesteps 50
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
gsutil cp data/outputs/lq_euler_50/checkpoint-4000 gs://vid_gen/runlong_temp_folder_for_pandas70m_debugging/fastvid/lq_euler_50/checkpoint-4000
|
||||
|
||||
|
||||
@@ -1,49 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 4\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=2\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 8 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--ema_decay 0.999\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thresh0.05_bs32"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "2.5,3.5,4.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.05
|
||||
@@ -1,45 +0,0 @@
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export NCCL_DEBUG=INFO
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=[MASTER_NODE_IP_ADDRESS]:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path data/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "data/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "data/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-7\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 8 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--ema_decay 0.999\
|
||||
--log_validation\
|
||||
--output_dir="data/outputs/lq_euler_50_thresh0.05_lr_1e-7"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "2.5,3.5,4.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.05
|
||||
@@ -1,44 +0,0 @@
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export NCCL_DEBUG=INFO
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=[MASTER_NODE_IP_ADDRESS]:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path data/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "data/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "data/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 8 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--ema_decay 0.999\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/shift1_euler_50"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--shift 1 \
|
||||
--validation_guidance_scale "2.5,3.5,4.5" \
|
||||
--num_euler_timesteps 50
|
||||
@@ -1,51 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 8 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thres0.1_lrg_0.75_phase1"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5,2.5,4.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--multi_phased_distill_schedule "4000-1"
|
||||
|
||||
@@ -1,52 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 8 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--ema_decay 0.999\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thres0.1_lrg_0.75_cfg_4.5"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5,2.5,4.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--distill_cfg 4.5
|
||||
|
||||
@@ -1,51 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 8 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--ema_decay 0.999\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thres0.15_lrg_0.75"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5,2.5,4.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.15 \
|
||||
--linear_range 0.75
|
||||
|
||||
@@ -1,54 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 8 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thres0.1_lrg_0.75_ema_0.95_decay0.0"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5,2.5,4.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--use_ema \
|
||||
--ema_decay 0.95 \
|
||||
--weight_decay 0.0
|
||||
|
||||
|
||||
@@ -1,57 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=offline
|
||||
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
CACHE_DIR=/data/.cache
|
||||
EXPERIMENT=lq_euler_50_thres0.1_lrg_0.75_cfg0.0
|
||||
OUTPUT_DIR=$DATA_DIR/outputs/$EXPERIMENT
|
||||
export WANDB_DIR=$DATA_DIR/wandb/
|
||||
|
||||
torchrun --nnodes 1 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir $CACHE_DIR \
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 1\
|
||||
--train_sp_batch_size 1\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=2000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 8 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--ema_decay 0.999\
|
||||
--log_validation\
|
||||
--output_dir=$OUTPUT_DIR \
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "1.5,2.5,4.5,6.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--not_apply_cfg_solver
|
||||
|
||||
@@ -1,55 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=offline
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
CACHE_DIR=/data/.cache
|
||||
EXPERIMENT=lq_euler_50_thres0.1_lrg_0.75_cfg0.0
|
||||
OUTPUT_DIR=$DATA_DIR/outputs/$EXPERIMENT
|
||||
export WANDB_DIR=$DATA_DIR/wandb/
|
||||
torchrun --nnodes 8 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir $CACHE_DIR \
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 24\
|
||||
--sp_size 8\
|
||||
--train_sp_batch_size 1\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=2000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 8 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--ema_decay 0.999\
|
||||
--log_validation\
|
||||
--output_dir=$OUTPUT_DIR \
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 139 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "1.5,2.5,4.5,6.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--not_apply_cfg_solver
|
||||
|
||||
@@ -1,52 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=offline
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
CACHE_DIR=/data/.cache
|
||||
EXPERIMENT=shift16_euler_50
|
||||
OUTPUT_DIR=$DATA_DIR/outputs/$EXPERIMENT
|
||||
export WANDB_DIR=$DATA_DIR/wandb/
|
||||
torchrun --nnodes 1 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir $CACHE_DIR \
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 1\
|
||||
--train_sp_batch_size 1\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=2000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 8 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--ema_decay 0.999\
|
||||
--log_validation\
|
||||
--output_dir=$OUTPUT_DIR \
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--shift 16 \
|
||||
--validation_guidance_scale "0.5,1.5,2.5,4.5" \
|
||||
--num_euler_timesteps 50
|
||||
|
||||
@@ -1,52 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=offline
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
CACHE_DIR=/data/.cache
|
||||
EXPERIMENT=shift16_euler_50
|
||||
OUTPUT_DIR=$DATA_DIR/outputs/$EXPERIMENT
|
||||
export WANDB_DIR=$DATA_DIR/wandb/
|
||||
torchrun --nnodes 8 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir $CACHE_DIR \
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 24\
|
||||
--sp_size 8\
|
||||
--train_sp_batch_size 1\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=2000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 8 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--ema_decay 0.999\
|
||||
--log_validation\
|
||||
--output_dir=$OUTPUT_DIR \
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 139 \
|
||||
--shift 16 \
|
||||
--validation_guidance_scale "0.5,1.5,2.5,4.5" \
|
||||
--num_euler_timesteps 50
|
||||
|
||||
@@ -1,52 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=offline
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
CACHE_DIR=/data/.cache
|
||||
EXPERIMENT=4step_infer_shift16_euler_50
|
||||
OUTPUT_DIR=$DATA_DIR/outputs/$EXPERIMENT
|
||||
export WANDB_DIR=$DATA_DIR/wandb/
|
||||
torchrun --nnodes 1 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir $CACHE_DIR \
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 1\
|
||||
--train_sp_batch_size 1\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=2000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 4 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--ema_decay 0.999\
|
||||
--log_validation\
|
||||
--output_dir=$OUTPUT_DIR \
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--shift 16 \
|
||||
--validation_guidance_scale "0.5,1.5,2.5,4.5" \
|
||||
--num_euler_timesteps 50
|
||||
|
||||
@@ -1,52 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=offline
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
CACHE_DIR=/data/.cache
|
||||
EXPERIMENT=4step_infer_shift16_euler_50
|
||||
OUTPUT_DIR=$DATA_DIR/outputs/$EXPERIMENT
|
||||
export WANDB_DIR=$DATA_DIR/wandb/
|
||||
torchrun --nnodes 8 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir $CACHE_DIR \
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 24\
|
||||
--sp_size 8\
|
||||
--train_sp_batch_size 1\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=2000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 4 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--ema_decay 0.999\
|
||||
--log_validation\
|
||||
--output_dir=$OUTPUT_DIR \
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 139 \
|
||||
--shift 16 \
|
||||
--validation_guidance_scale "0.5,1.5,2.5,4.5" \
|
||||
--num_euler_timesteps 50
|
||||
|
||||
@@ -1,48 +0,0 @@
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export NCCL_DEBUG=INFO
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=[MASTER_NODE_IP_ADDRESS]:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path data/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "data/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "data/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--uncond_prompt_dir "data/validation_embeddings/uncond_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 250\
|
||||
--validation_sampling_steps 8 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--ema_decay 0.999\
|
||||
--log_validation\
|
||||
--output_dir="data/outputs/lq_euler_50_thresh0.05"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale 4.5 \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.05
|
||||
|
||||
gsutil cp data/outputs/lq_euler_50_thresh0.05/checkpoint-4000 gs://vid_gen/runlong_temp_folder_for_pandas70m_debugging/fastvid/lq_euler_50_thresh0.05/checkpoint-4000
|
||||
@@ -1,52 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=offline
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
CACHE_DIR=/data/.cache
|
||||
EXPERIMENT=4step_infer_shift12_euler_50
|
||||
OUTPUT_DIR=$DATA_DIR/outputs/$EXPERIMENT
|
||||
export WANDB_DIR=$DATA_DIR/wandb/
|
||||
torchrun --nnodes 1 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir $CACHE_DIR \
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 1\
|
||||
--train_sp_batch_size 1\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=2000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 4 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--ema_decay 0.999\
|
||||
--log_validation\
|
||||
--output_dir=$OUTPUT_DIR \
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--shift 12 \
|
||||
--validation_guidance_scale "0.5,1.5,2.5,4.5" \
|
||||
--num_euler_timesteps 50
|
||||
|
||||
@@ -1,52 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=offline
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
CACHE_DIR=/data/.cache
|
||||
EXPERIMENT=4step_infer_shift12_euler_50
|
||||
OUTPUT_DIR=$DATA_DIR/outputs/$EXPERIMENT
|
||||
export WANDB_DIR=$DATA_DIR/wandb/
|
||||
torchrun --nnodes 8 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir $CACHE_DIR \
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 24\
|
||||
--sp_size 8\
|
||||
--train_sp_batch_size 1\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=2000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 4 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--ema_decay 0.999\
|
||||
--log_validation\
|
||||
--output_dir=$OUTPUT_DIR \
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 139 \
|
||||
--shift 12 \
|
||||
--validation_guidance_scale "0.5,1.5,2.5,4.5" \
|
||||
--num_euler_timesteps 50
|
||||
|
||||
@@ -1,54 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=offline
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
CACHE_DIR=/data/.cache
|
||||
EXPERIMENT=4step_infer_lq_euler_50_thresh0.1_lrg_0.75
|
||||
OUTPUT_DIR=$DATA_DIR/outputs/$EXPERIMENT
|
||||
export WANDB_DIR=$DATA_DIR/wandb/
|
||||
torchrun --nnodes 1 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir $CACHE_DIR \
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 1\
|
||||
--train_sp_batch_size 1\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=2000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 4 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--ema_decay 0.999\
|
||||
--log_validation\
|
||||
--output_dir=$OUTPUT_DIR \
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "1.5,2.5,4.5,6.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75
|
||||
|
||||
@@ -1,54 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=offline
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
CACHE_DIR=/data/.cache
|
||||
EXPERIMENT=4step_infer_lq_euler_50_thresh0.1_lrg_0.75
|
||||
OUTPUT_DIR=$DATA_DIR/outputs/$EXPERIMENT
|
||||
export WANDB_DIR=$DATA_DIR/wandb/
|
||||
torchrun --nnodes 8 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir $CACHE_DIR \
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 24\
|
||||
--sp_size 8\
|
||||
--train_sp_batch_size 1\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=2000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 4 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--ema_decay 0.999\
|
||||
--log_validation\
|
||||
--output_dir=$OUTPUT_DIR \
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 139 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "1.5,2.5,4.5,6.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75
|
||||
|
||||
@@ -1,55 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=offline
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
CACHE_DIR=/data/.cache
|
||||
EXPERIMENT=lq_euler_50_thresh0.1_lrg_0.75_phase1
|
||||
OUTPUT_DIR=$DATA_DIR/outputs/$EXPERIMENT
|
||||
export WANDB_DIR=$DATA_DIR/wandb/
|
||||
torchrun --nnodes 1 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir $CACHE_DIR \
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 1\
|
||||
--train_sp_batch_size 1\
|
||||
--dataloader_num_workers 8\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=2000\
|
||||
--learning_rate=1e-7\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 4 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--ema_decay 0.999\
|
||||
--log_validation\
|
||||
--output_dir=$OUTPUT_DIR \
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "1.5,2.5,4.5,6.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--multi_phased_distill_schedule "4000-1"
|
||||
|
||||
@@ -1,55 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=offline
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
CACHE_DIR=/data/.cache
|
||||
EXPERIMENT=lq_euler_50_thresh0.1_lrg_0.75_phase1
|
||||
OUTPUT_DIR=$DATA_DIR/outputs/$EXPERIMENT
|
||||
export WANDB_DIR=$DATA_DIR/wandb/
|
||||
torchrun --nnodes 8 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir $CACHE_DIR \
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 24\
|
||||
--sp_size 8\
|
||||
--train_sp_batch_size 1\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=2000\
|
||||
--learning_rate=1e-7\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 8 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--ema_decay 0.999\
|
||||
--log_validation\
|
||||
--output_dir=$OUTPUT_DIR \
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 139 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "1.5,2.5,4.5,6.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--multi_phased_distill_schedule "4000-1"
|
||||
|
||||
@@ -1,52 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=offline
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
CACHE_DIR=/data/.cache
|
||||
EXPERIMENT=lq_euler_50_thres0.1_lrg_0.75_bs_64
|
||||
OUTPUT_DIR=$DATA_DIR/outputs/$EXPERIMENT
|
||||
export WANDB_DIR=$DATA_DIR/wandb/
|
||||
export WANDB_DIR=$DATA_DIR/wandb/
|
||||
torchrun --nnodes 8 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir $CACHE_DIR \
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 1\
|
||||
--train_sp_batch_size 1\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=2000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 8 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir=$OUTPUT_DIR \
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5,2.5,4.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1
|
||||
@@ -1,52 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=offline
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
CACHE_DIR=/data/.cache
|
||||
EXPERIMENT=4step_infer_shift16_euler_50
|
||||
OUTPUT_DIR=$DATA_DIR/outputs/$EXPERIMENT
|
||||
export WANDB_DIR=$DATA_DIR/wandb/
|
||||
|
||||
torchrun --nnodes 1 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir $CACHE_DIR \
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 1\
|
||||
--train_sp_batch_size 1\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=2000\
|
||||
--learning_rate=5e-7\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 8 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir=$OUTPUT_DIR \
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5,2.5,4.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1
|
||||
@@ -1,51 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=offline
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
CACHE_DIR=/data/.cache
|
||||
EXPERIMENT=4step_infer_shift16_euler_50
|
||||
OUTPUT_DIR=$DATA_DIR/outputs/$EXPERIMENT
|
||||
export WANDB_DIR=$DATA_DIR/wandb/
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir $CACHE_DIR \
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 24\
|
||||
--sp_size 8\
|
||||
--train_sp_batch_size 1\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=2000\
|
||||
--learning_rate=5e-7\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 8 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir=$OUTPUT_DIR \
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 139 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5,2.5,4.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1
|
||||
@@ -1,49 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=data
|
||||
IP=172.23.30.16
|
||||
|
||||
torchrun --nnodes 4 --nproc_per_node 4\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 1\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 8 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/shift1_euler_50_0.75_phase1"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--shift 1 \
|
||||
--validation_guidance_scale "0.5,1.5,2.5,4.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--multi_phased_distill_schedule "4000-1"
|
||||
|
||||
@@ -1,43 +0,0 @@
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
|
||||
|
||||
DATA_DIR=data
|
||||
|
||||
torchrun --nnodes 1 --nproc_per_node 8\
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 1\
|
||||
--train_sp_batch_size 1\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 8 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thres0.1_lrg_0.75_phase_ema0.95"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5,2.5,4.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--multi_phased_distill_schedule "4000-1" \
|
||||
--use_ema \
|
||||
--ema_decay 0.95
|
||||
|
||||
@@ -1,49 +0,0 @@
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
|
||||
|
||||
DATA_DIR=data
|
||||
IP=172.23.30.16
|
||||
|
||||
torchrun --nnodes 4 --nproc_per_node 4\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 2\
|
||||
--train_sp_batch_size 1\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 8 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thres0.1_lrg_0.75_phase_ema0.95_cfg4.5"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5,2.5,4.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--multi_phased_distill_schedule "4000-1" \
|
||||
--use_ema \
|
||||
--ema_decay 0.95 \
|
||||
--distill_cfg 4.5
|
||||
|
||||
@@ -1,53 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 2\
|
||||
--train_sp_batch_size 1\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps "4,8" \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thresh0.1_lrg_0.75_phase1_ema0.95"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5,2.5,4.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--multi_phased_distill_schedule "4000-1" \
|
||||
--use_ema \
|
||||
--ema_decay 0.95
|
||||
|
||||
@@ -1,54 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 2\
|
||||
--train_sp_batch_size 1\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps "4,8" \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thres0.1_lrg_0.75_phase_ema0.95_cfg6.0"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--multi_phased_distill_schedule "4000-1" \
|
||||
--use_ema \
|
||||
--ema_decay 0.95 \
|
||||
--distill_cfg 6.0
|
||||
|
||||
@@ -1,47 +0,0 @@
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export NCCL_DEBUG=INFO
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=[MASTER_NODE_IP_ADDRESS]:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path data/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "data/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "data/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--uncond_prompt_dir "data/validation_embeddings/uncond_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 8 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--ema_decay 0.999\
|
||||
--log_validation\
|
||||
--output_dir="data/outputs/shift8_euler_100"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--shift 8.0 \
|
||||
--validation_guidance_scale 4.5
|
||||
|
||||
|
||||
gsutil cp data/outputs/shift8_euler_100/checkpoint-4000 gs://vid_gen/runlong_temp_folder_for_pandas70m_debugging/fastvid/shift8_euler_100/checkpoint-4000
|
||||
@@ -1,54 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 2\
|
||||
--train_sp_batch_size 1\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps "4,8" \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thresh0.1_lrg_0.75_phase1_ema0.98_cfg4.5"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--multi_phased_distill_schedule "4000-1" \
|
||||
--use_ema \
|
||||
--ema_decay 0.98 \
|
||||
--distill_cfg 4.5
|
||||
|
||||
@@ -1,51 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 2\
|
||||
--train_sp_batch_size 1\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=3e-7\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps "4,8" \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thresh0.1_lrg_0.75_phase1_lr_3e-7"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--multi_phased_distill_schedule "4000-1"
|
||||
|
||||
@@ -1,54 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 2\
|
||||
--train_sp_batch_size 1\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps "4,8" \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thresh0.15_lrg_0.75_phase1_ema0.95_cfg4.5"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.15 \
|
||||
--linear_range 0.75 \
|
||||
--multi_phased_distill_schedule "4000-1" \
|
||||
--use_ema \
|
||||
--ema_decay 0.95 \
|
||||
--distill_cfg 4.5
|
||||
|
||||
@@ -1,53 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.142.161
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps "8" \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--ema_decay 0.999\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thres0.1_linear_range_0.75_cfg_7.0"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5,2.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--distill_cfg 6.0 \
|
||||
--multi_phased_distill_schedule "4000-8" \
|
||||
|
||||
@@ -1,46 +0,0 @@
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
|
||||
|
||||
DATA_DIR=data
|
||||
IP=172.23.30.16
|
||||
|
||||
torchrun --nnodes 4 --nproc_per_node 4\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 2\
|
||||
--train_sp_batch_size 1\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=125\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps "8" \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thres0.1_lrg_0.75_reproduce"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5,2.5,4.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--multi_phased_distill_schedule "4000-8" \
|
||||
|
||||
@@ -1,51 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=5e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=64\
|
||||
--validation_steps 64\
|
||||
--validation_sampling_steps "8" \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thres0.1_lrg_0.75_phase1_lr5e-6"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5,2.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--multi_phased_distill_schedule "4000-1"
|
||||
|
||||
@@ -1,51 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=64\
|
||||
--validation_steps 64\
|
||||
--validation_sampling_steps "8" \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thres0.1_lrg_0.75_phase2_lr1e-6"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5,2.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--multi_phased_distill_schedule "4000-2"
|
||||
|
||||
@@ -1,51 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=5e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=64\
|
||||
--validation_steps 64\
|
||||
--validation_sampling_steps "8" \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thres0.1_lrg_0.75_phase2_lr5e-6"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5,2.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--multi_phased_distill_schedule "4000-2"
|
||||
|
||||
@@ -1,51 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=5e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=64\
|
||||
--validation_steps 64\
|
||||
--validation_sampling_steps "8" \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thres0.1_lrg_0.75_phase8_lr5e-6"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5,2.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--multi_phased_distill_schedule "4000-8"
|
||||
|
||||
@@ -1,51 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-5\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=64\
|
||||
--validation_steps 64\
|
||||
--validation_sampling_steps "8" \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thres0.1_lrg_0.75_phase8_lr1e-5"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5,2.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--multi_phased_distill_schedule "4000-8"
|
||||
|
||||
@@ -1,49 +0,0 @@
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export NCCL_DEBUG=INFO
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=[MASTER_NODE_IP_ADDRESS]:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path data/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "data/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "data/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--uncond_prompt_dir "data/validation_embeddings/uncond_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 8 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--ema_decay 0.999\
|
||||
--log_validation\
|
||||
--output_dir="data/outputs/shift8_euler_50"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--shift 8.0 \
|
||||
--validation_guidance_scale 4.5 \
|
||||
--num_euler_timesteps 50
|
||||
|
||||
gsutil cp data/outputs/shift8_euler_50/checkpoint-4000 gs://vid_gen/runlong_temp_folder_for_pandas70m_debugging/fastvid/shift8_euler_50/checkpoint-4000
|
||||
|
||||
|
||||
@@ -1,51 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=64\
|
||||
--validation_steps 64\
|
||||
--validation_sampling_steps "8" \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thres0.1_lrg_0.75_phase1_lr1e-6_repro"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5,2.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--multi_phased_distill_schedule "4000-1"
|
||||
|
||||
@@ -1,52 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=125\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps "8" \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thres0.1_lrg_0.75_reproduce"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5,2.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--multi_phased_distill_schedule "4000-8" \
|
||||
|
||||
@@ -1,51 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=64\
|
||||
--validation_steps 64\
|
||||
--validation_sampling_steps "4,8" \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thres0.1_lrg_0.75_phase4_lr1e-6"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5,2.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--multi_phased_distill_schedule "4000-4"
|
||||
|
||||
@@ -1,52 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=64\
|
||||
--validation_steps 64\
|
||||
--validation_sampling_steps "4,8" \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thres0.1_lrg_0.75_phase1_lr1e-6_cfg6.0"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5,2.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--multi_phased_distill_schedule "4000-1" \
|
||||
--distill_cfg 6.0
|
||||
|
||||
@@ -1,46 +0,0 @@
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
|
||||
|
||||
DATA_DIR=data
|
||||
IP=172.23.30.16
|
||||
|
||||
torchrun --nnodes 4 --nproc_per_node 4\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 2\
|
||||
--train_sp_batch_size 1\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=5e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps "8" \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thres0.1_lrg_0.75_phase1_lr_5e-6_test_norm"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--multi_phased_distill_schedule "4000-1"
|
||||
|
||||
@@ -1,47 +0,0 @@
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
|
||||
|
||||
DATA_DIR=data
|
||||
IP=172.23.30.16
|
||||
|
||||
torchrun --nnodes 4 --nproc_per_node 4\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 14\
|
||||
--sp_size 2\
|
||||
--train_sp_batch_size 1\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=5e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 64\
|
||||
--validation_sampling_steps "8" \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thres0.1_lrg_0.75_phase1_lr_5e-6_pred_decay_0.1_latent14"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--multi_phased_distill_schedule "4000-1" \
|
||||
--pred_decay_weight 0.1
|
||||
|
||||
@@ -1,136 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=640\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=128\
|
||||
--validation_steps 64\
|
||||
--validation_sampling_steps "8" \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thres0.1_lrg_0.75_phase1_lr1e-6_pred_decay0.1"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5,2.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--multi_phased_distill_schedule "4000-1" \
|
||||
--pred_decay_weight 0.1
|
||||
|
||||
|
||||
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=640\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=128\
|
||||
--validation_steps 64\
|
||||
--validation_sampling_steps "8" \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thres0.1_lrg_0.75_phase1_lr1e-6_pred_decay0.1_l2"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5,2.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--multi_phased_distill_schedule "4000-1" \
|
||||
--pred_decay_weight 0.1 \
|
||||
--pred_decay_type "l2"
|
||||
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=640\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=128\
|
||||
--validation_steps 64\
|
||||
--validation_sampling_steps "8" \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thres0.1_lrg_0.75_phase1_lr1e-6_pred_decay0.001"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5,2.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--multi_phased_distill_schedule "4000-1" \
|
||||
--pred_decay_weight 0.001
|
||||
@@ -1,135 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=640\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=128\
|
||||
--validation_steps 64\
|
||||
--validation_sampling_steps "8" \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thres0.1_lrg_0.75_phase1_lr1e-6_pred_decay0.05"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5,2.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--multi_phased_distill_schedule "4000-1" \
|
||||
--pred_decay_weight 0.05
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=640\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=128\
|
||||
--validation_steps 64\
|
||||
--validation_sampling_steps "8" \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thres0.1_lrg_0.75_phase1_lr1e-6_pred_decay0.05_l2"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5,2.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--multi_phased_distill_schedule "4000-1" \
|
||||
--pred_decay_weight 0.05 \
|
||||
--pred_decay_type "l2"
|
||||
|
||||
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=640\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=128\
|
||||
--validation_steps 64\
|
||||
--validation_sampling_steps "8" \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thres0.1_lrg_0.75_phase1_lr1e-6_pred_decay0.001_l2"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5,2.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--multi_phased_distill_schedule "4000-1" \
|
||||
--pred_decay_weight 0.001 \
|
||||
--pred_decay_type "l2"
|
||||
@@ -1,136 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.139.86
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=1000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=128\
|
||||
--validation_steps 64\
|
||||
--validation_sampling_steps "8" \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thres0.1_lrg_0.75_phase1_lr1e-6_pred_decay0.01"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5,2.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--multi_phased_distill_schedule "4000-1" \
|
||||
--pred_decay_weight 0.01
|
||||
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=640\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=128\
|
||||
--validation_steps 64\
|
||||
--validation_sampling_steps "8" \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thres0.1_lrg_0.75_phase1_lr1e-6_pred_decay0.01_l2"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5,2.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--multi_phased_distill_schedule "4000-1" \
|
||||
--pred_decay_weight 0.01 \
|
||||
--pred_decay_type "l2"
|
||||
|
||||
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=640\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=128\
|
||||
--validation_steps 64\
|
||||
--validation_sampling_steps "8" \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thres0.1_lrg_0.75_phase1_lr1e-6_pred_decay0.005_l2"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "0.5,1.5,2.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75 \
|
||||
--multi_phased_distill_schedule "4000-1" \
|
||||
--pred_decay_weight 0.005 \
|
||||
--pred_decay_type "l2"
|
||||
@@ -1,49 +0,0 @@
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
# This can avoid OOM
|
||||
export PYTORCH_CUDA_ALLOC_CONF='max_split_size_mb:1024'
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export NCCL_DEBUG=INFO
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=[MASTER_NODE_IP_ADDRESS]:29500 \
|
||||
fastvideo/distill_adv.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path data/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "data/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "data/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--uncond_prompt_dir "data/validation_embeddings/uncond_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=250\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 8 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--ema_decay 0.999\
|
||||
--log_validation\
|
||||
--output_dir="data/outputs"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--shift 8.0 \
|
||||
--validation_guidance_scale 4.5
|
||||
|
||||
|
||||
gsutil cp data/outputs/adv_shift8_euler_100/checkpoint-4000 gs://vid_gen/runlong_temp_folder_for_pandas70m_debugging/fastvid/adv_shift8_euler_100/checkpoint-4000
|
||||
@@ -1,52 +0,0 @@
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
# This can avoid OOM
|
||||
export PYTORCH_CUDA_ALLOC_CONF='max_split_size_mb:1024'
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export NCCL_DEBUG=INFO
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=[MASTER_NODE_IP_ADDRESS]:29500 \
|
||||
fastvideo/distill_adv.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path data/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "data/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "data/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--uncond_prompt_dir "data/validation_embeddings/uncond_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=250\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 8 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--ema_decay 0.999\
|
||||
--log_validation\
|
||||
--output_dir="data/outputs/adv_lq_euler_50"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale 4.5 \
|
||||
--num_euler_timesteps 50
|
||||
|
||||
gsutil cp data/outputs/adv_lq_euler_50/checkpoint-4000 gs://vid_gen/runlong_temp_folder_for_pandas70m_debugging/fastvid/adv_lq_euler_50/checkpoint-4000
|
||||
|
||||
|
||||
@@ -1,51 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.131.23
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 8 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--ema_decay 0.999\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thresh0.05_multiphaseshedule_125-16,250-8,375-4,500-2"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "2.5,3.5,4.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.05 \
|
||||
--multi_phased_distill_schedule "125-16,250-8,375-4,500-2"
|
||||
|
||||
@@ -1,51 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.142.161
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 8 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--ema_decay 0.999\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thres0.1_linear_range_0.75"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "2.5,3.5,4.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.1 \
|
||||
--linear_range 0.75
|
||||
|
||||
@@ -1,51 +0,0 @@
|
||||
export WANDB_BASE_URL="https://api.wandb.ai"
|
||||
export WANDB_DIR="$HOME"
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
export LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/aws-ofi-nccl/lib:$LD_LIBRARY_PATH
|
||||
export FI_PROVIDER=efa
|
||||
export FI_EFA_USE_DEVICE_RDMA=1
|
||||
export NCCL_PROTO=simple
|
||||
|
||||
DATA_DIR=/data
|
||||
IP=10.4.129.100
|
||||
|
||||
torchrun --nnodes 2 --nproc_per_node 8\
|
||||
--node_rank=0 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=$IP:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path $DATA_DIR/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "$DATA_DIR/Merge-30k-Data/video2caption.json"\
|
||||
--validation_prompt_dir "$DATA_DIR/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 8 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--ema_decay 0.999\
|
||||
--log_validation\
|
||||
--output_dir="$DATA_DIR/outputs/lq_euler_50_thres0.05_linear_range_0.75"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale "2.5,3.5,4.5" \
|
||||
--num_euler_timesteps 50 \
|
||||
--linear_quadratic_threshold 0.05 \
|
||||
--linear_range 0.75
|
||||
|
||||
@@ -34,84 +34,3 @@ torchrun --nnodes 4 --nproc_per_node 4 \
|
||||
--num_frames 91 \
|
||||
--selective_checkpointing 1.0
|
||||
|
||||
|
||||
torchrun --nnodes 4 --nproc_per_node 4 \
|
||||
--node_rank=3 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=172.23.30.15:29500 \
|
||||
fastvideo/train.py \
|
||||
--seed 42 \
|
||||
--pretrained_model_name_or_path data/mochi \
|
||||
--cache_dir "data/.cache" \
|
||||
--data_json_path "data/BLACK-MYTH-DREAM-BOOTH/videos2caption.json" \
|
||||
--validation_prompt_dir "data/BLACK-MYTH-DREAM-BOOTH/validation_prompt_embed_mask" \
|
||||
--uncond_prompt_dir "data/BLACK-MYTH-DREAM-BOOTH/uncond_prompt_embed_mask" \
|
||||
--gradient_checkpointing \
|
||||
--train_batch_size=1 \
|
||||
--num_latent_t 16 \
|
||||
--sp_size 4 \
|
||||
--train_sp_batch_size 4 \
|
||||
--dataloader_num_workers 4 \
|
||||
--gradient_accumulation_steps=1 \
|
||||
--max_train_steps=4000 \
|
||||
--learning_rate=2e-5 \
|
||||
--mixed_precision="bf16" \
|
||||
--checkpointing_steps=100 \
|
||||
--validation_steps=50 \
|
||||
--validation_sampling_steps 64 \
|
||||
--checkpoints_total_limit 3 \
|
||||
--allow_tf32 \
|
||||
--ema_start_step 0 \
|
||||
--cfg 0.1 \
|
||||
--ema_decay 0.999 \
|
||||
--log_validation \
|
||||
--output_dir="data/outputs/black_myth_correct_video_mask" \
|
||||
--weighting_scheme "uniform" \
|
||||
--num_frames 91 \
|
||||
--selective_checkpointing 1.0
|
||||
|
||||
|
||||
|
||||
|
||||
torchrun --nnodes 1 --nproc_per_node 4 \
|
||||
fastvideo/train.py \
|
||||
--seed 42 \
|
||||
--pretrained_model_name_or_path data/mochi \
|
||||
--cache_dir "data/.cache" \
|
||||
--data_json_path "data/BLACK-MYTH-Finetune-Dataset/videos2caption.json" \
|
||||
--validation_prompt_dir "data/BLACK-MYTH-Finetune-Dataset/validation_prompt_embed_mask" \
|
||||
--uncond_prompt_dir "data/BLACK-MYTH-Finetune-Dataset/uncond_prompt_embed_mask" \
|
||||
--gradient_checkpointing \
|
||||
--train_batch_size=1 \
|
||||
--num_latent_t 16 \
|
||||
--sp_size 4 \
|
||||
--train_sp_batch_size 4 \
|
||||
--dataloader_num_workers 4 \
|
||||
--gradient_accumulation_steps=1 \
|
||||
--max_train_steps=2000 \
|
||||
--learning_rate=2e-5 \
|
||||
--mixed_precision="bf16" \
|
||||
--checkpointing_steps=100 \
|
||||
--validation_steps=100 \
|
||||
--validation_sampling_steps 64 \
|
||||
--checkpoints_total_limit 3 \
|
||||
--allow_tf32 \
|
||||
--ema_start_step 0 \
|
||||
--cfg 0.0 \
|
||||
--ema_decay 0.999 \
|
||||
--log_validation \
|
||||
--output_dir="data/outputs/black_myth_dream" \
|
||||
--weighting_scheme "uniform" \
|
||||
--num_frames 91 \
|
||||
|
||||
|
||||
| 4/2000 [01:23<10:59:33, 19.83s/it, loss=0.74, grad_norm=2.6]
|
||||
|
||||
|
||||
| 2/2000 [00:44<11:59:27, 21.61s/it, loss=0.873, grad_norm=1.57]
|
||||
| 3/2000 [01:03<11:18:51, 20.40s/it, loss=0.813, grad_norm=1.48]
|
||||
|
||||
|
||||
# FSDP Hybrid:
|
||||
|
||||
|
||||
@@ -1,39 +0,0 @@
|
||||
accelerate launch \
|
||||
--config_file scripts/accelerate_configs/deepspeed_zero2_config.yaml \
|
||||
fastvideo/train.py \
|
||||
--seed 42 \
|
||||
--pretrained_model_name_or_path data/mochi \
|
||||
--cache_dir "data/.cache" \
|
||||
--data_merge_path "data/Mochi-Synthetic-Data/merge.txt" \
|
||||
--gradient_checkpointing \
|
||||
--train_batch_size=1 \
|
||||
--num_latent_t 14 \
|
||||
--dataloader_num_workers 1 \
|
||||
--gradient_accumulation_steps=4 \
|
||||
--max_train_steps=200 \
|
||||
--learning_rate=1e-5 \
|
||||
--lr_scheduler="constant" \
|
||||
--lr_warmup_steps=0 \
|
||||
--mixed_precision="bf16" \
|
||||
--checkpointing_steps=50 \
|
||||
--allow_tf32 \
|
||||
--tile_sample_stride 192 \
|
||||
--ema_start_step 0 \
|
||||
--cfg 0.1 \
|
||||
--ema_decay 0.999 \
|
||||
--enable_tiling \
|
||||
--sp_size 1 \
|
||||
--train_sp_batch_size 1 \
|
||||
--output_dir="data/outputs/debug_1e-5"
|
||||
|
||||
python fastvideo/sample/sample_t2v_mochi.py \
|
||||
--model_path data/mochi \
|
||||
--prompt "A hand enters the frame, pulling a sheet of plastic wrap over three balls of dough placed on a wooden surface. The plastic wrap is stretched to cover the dough more securely. The hand adjusts the wrap, ensuring that it is tight and smooth over the dough. The scene focuses on the hand's movements as it secures the edges of the plastic wrap. No new objects appear, and the camera remains stationary, focusing on the action of covering the dough." \
|
||||
--num_frames 79 \
|
||||
--height 480 \
|
||||
--width 848 \
|
||||
--num_inference_steps 64 \
|
||||
--guidance_scale 4.5 \
|
||||
--seed 12346 \
|
||||
--transformer_path data/outputs/debug_1e-5/checkpoint-50/transformer \
|
||||
--output_path 123.mp4
|
||||
@@ -1,80 +0,0 @@
|
||||
|
||||
export WANDB_MODE=online
|
||||
export WANDB_API_KEY=4f6de3765d6464f43e0506ec7d785641af645e73
|
||||
|
||||
torchrun --nnodes 4 --nproc_per_node 4\
|
||||
--node_rank=3 \
|
||||
--rdzv_id=456 \
|
||||
--rdzv_backend=c10d \
|
||||
--rdzv_endpoint=172.23.30.16:29500 \
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path data/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "data/General-Video/videos2caption.json"\
|
||||
--validation_prompt_dir "data/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--uncond_prompt_dir "data/validation_embeddings/uncond_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 28\
|
||||
--sp_size 4\
|
||||
--train_sp_batch_size 2\
|
||||
--dataloader_num_workers 4\
|
||||
--gradient_accumulation_steps=1\
|
||||
--max_train_steps=4000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=500\
|
||||
--validation_steps 125\
|
||||
--validation_sampling_steps 8 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--ema_decay 0.999\
|
||||
--log_validation\
|
||||
--output_dir="data/outputs/shift8_euler_50_olddata_no_ema"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 163 \
|
||||
--shift 8.0 \
|
||||
--validation_guidance_scale 4.5 \
|
||||
--num_euler_timesteps 50
|
||||
|
||||
|
||||
|
||||
|
||||
export WANDB_MODE=online
|
||||
|
||||
torchrun --nnodes 1 --nproc_per_node 4\
|
||||
fastvideo/distill.py\
|
||||
--seed 42\
|
||||
--pretrained_model_name_or_path data/mochi\
|
||||
--cache_dir "data/.cache"\
|
||||
--data_json_path "data/Image-Train-Dataset/videos2caption.json"\
|
||||
--validation_prompt_dir "data/validation_embeddings/validation_prompt_embed_mask"\
|
||||
--uncond_prompt_dir "data/validation_embeddings/uncond_prompt_embed_mask"\
|
||||
--gradient_checkpointing\
|
||||
--train_batch_size=1\
|
||||
--num_latent_t 1\
|
||||
--sp_size 1\
|
||||
--train_sp_batch_size 1\
|
||||
--dataloader_num_workers 4\
|
||||
--max_train_steps=20000\
|
||||
--learning_rate=1e-6\
|
||||
--mixed_precision="bf16"\
|
||||
--checkpointing_steps=50\
|
||||
--validation_steps 2\
|
||||
--validation_sampling_steps 8 \
|
||||
--checkpoints_total_limit 3\
|
||||
--allow_tf32\
|
||||
--ema_start_step 0\
|
||||
--cfg 0.0\
|
||||
--ema_decay 0.999\
|
||||
--log_validation\
|
||||
--output_dir="data/outputs/video_distill_shift_8_precision_debugged"\
|
||||
--tracker_project_name PCM \
|
||||
--num_frames 67 \
|
||||
--scheduler_type pcm_linear_quadratic \
|
||||
--validation_guidance_scale 4.5 \
|
||||
--num_euler_timesteps 50 \
|
||||
--use_lora
|
||||
@@ -1,30 +0,0 @@
|
||||
# export WANDB_MODE="offline"
|
||||
|
||||
accelerate launch \
|
||||
--config_file scripts/accelerate_configs/single.yaml \
|
||||
fastvideo/train.py \
|
||||
--pretrained_model_name_or_path data/mochi \
|
||||
--cache_dir "data/.cache" \
|
||||
--data_merge_path "data/Mochi-Synthetic-Data/merge.txt" \
|
||||
--gradient_checkpointing \
|
||||
--train_batch_size=1 \
|
||||
--num_latent_t 14 \
|
||||
--dataloader_num_workers 1 \
|
||||
--gradient_accumulation_steps=1 \
|
||||
--max_train_steps=1000000 \
|
||||
--learning_rate=1e-4 \
|
||||
--lr_scheduler="constant" \
|
||||
--lr_warmup_steps=0 \
|
||||
--optimizer="AdamW" \
|
||||
--mixed_precision="bf16" \
|
||||
--checkpointing_steps=250 \
|
||||
--allow_tf32 \
|
||||
--tile_sample_stride 192 \
|
||||
--ema_start_step 0 \
|
||||
--cfg 0.1 \
|
||||
--resume_from_checkpoint="latest" \
|
||||
--ema_decay 0.999 \
|
||||
--enable_tiling \
|
||||
--sp_size 1 \
|
||||
--train_sp_batch_size 1 \
|
||||
--output_dir="data/outputs/debug" \
|
||||
Reference in New Issue
Block a user