Initial commit

Co-authored-by: Jensen Zhou <jensen.zhou@stability.ai> Co-authored-by: Aaryaman Vasishta <aaryaman.vasishta@stability.ai>
2025-03-18 07:49:36 -07:00
commit 8a640b7316
53 changed files with 8131 additions and 0 deletions
@@ -0,0 +1,76 @@
+# :rocket: Gradio Demo
+
+This gradio demo is the simplest starting point for you play with our project.
+
+You can either visit it at our huggingface space [here](https://huggingface.co/spaces/stabilityai/stable-virtual-camera) or run it locally yourself by
+
+```bash
+python demo_gr.py
+```
+
+We provide two ways to use our demo:
+
+1. `Basic` mode, where user can upload a single image, and set a target camera trajectory from our preset options. This is the most straightforward way to use our model, and is suitable for most users.
+2. `Advanced` mode, where user can upload one or multiple images, and set a target camera trajectory by interacting with a 3D viewport (powered by [viser](https://viser.studio/latest)). This is suitable for power users and academic researchers.
+
+### `Basic`
+
+This is the default mode when entering our demo (given its simplicity).
+
+User can upload a single image, and set a target camera trajectory from our preset options. This is the most straightforward way to use our model, and is suitable for most users.
+
+Here is a video walkthrough:
+
+https://github.com/user-attachments/assets/4d965fa6-d8eb-452c-b773-6e09c88ca705
+
+You can choose from 13 preset trajectories that are common for NVS (`move-forward/backward` are omitted for visualization purpose):
+
+https://github.com/user-attachments/assets/b2cf8700-3d85-44b9-8d52-248e82f1fb55
+
+More formally:
+
+- `orbit/spiral/lemniscate` are good for showing the "3D-ness" of the scene.
+- `zoom-in/out` keep the camera position the same while increasing/decreasing the focal length.
+- `dolly zoom-in/out` move camera position backward/forward while increasing/decreasing the focal length.
+- `move-forward/backward/up/down/left/right` move camera position in different directions.
+
+Notes:
+
+- For a 80 frame video at `786x576` resolution, it takes around 20 seconds for the first pass generation, and around 2 minutes for the second pass generation, tested with a single H100 GPU.
+- Please expect around ~2-3x more times on HF space.
+
+### `Advanced`
+
+This is the power mode where you can have very fine-grained control over camera trajectories.
+
+User can upload one or multiple images, and set a target camera trajectory by interacting with a 3D viewport. This is suitable for power users and academic researchers.
+
+Here is a video walkthrough
+
+https://github.com/user-attachments/assets/dcec1be0-bd10-441e-879c-d1c2b63091ba
+
+Notes:
+
+- For a 134 frame video at `576x576` resolution, it takes around 16 seconds for the first pass generation, and around 4 minutes for the second pass generation, tested with a single H100 GPU.
+- Please expect around ~2-3x more times on HF space.
+
+### Pro tips
+
+- If the first pass sampling result is bad, click "Abort rendering" button in GUI to avoid stucking at second pass sampling such that you can try something else.
+
+### Performance benchmark
+
+We have tested our gradio demo in both a local environment and the HF space environment, across different modes and compilation settings. Here are our results:
+| Total time (s) | `Basic` first pass | `Basic` second pass | `Advanced` first pass | `Advanced` second pass |
+|:------------------------:|:-----------------:|:------------------:|:--------------------:|:---------------------:|
+| HF (L40S, w/o comp.) | 68 | 484 | 48 | 780 |
+| HF (L40S, w/ comp.) | 51 | 362 | 36 | 587 |
+| Local (H100, w/o comp.) | 35 | 204 | 20 | 313 |
+| Local (H100, w/ comp.) | 21 | 144 | 16 | 234 |
+
+Notes:
+
+- HF space uses L40S GPU, and our local environment uses H100 GPU.
+- We opt-in compilation by `torch.compile`.
+- `Basic` mode is tested by generating 80 frames at `768x576` resolution.
+- `Advanced` mode is tested by generating 134 frames at `576x576` resolution.