nodes/Lightricks-ComfyUI-LTXVideo
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Automated PR - 2026-01-06

Browse Source
This commit is contained in:
sync-bot
2026-01-06 06:06:43 +00:00
parent 08f892eace
commit e1d2cff331
91 changed files with 34231 additions and 65642 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitattributes
+8
View File
@@ -1,2 +1,10 @@
**/*.png filter=lfs diff=lfs merge=lfs -text
**/*.mp4 filter=lfs diff=lfs merge=lfs -text
**/*.jpg filter=lfs diff=lfs merge=lfs -text
example_workflows/low_level/end.jpg -filter -diff -merge -text
example_workflows/low_level/fox.jpg -filter -diff -merge -text
example_workflows/low_level/shrek2.jpg -filter -diff -merge -text
example_workflows/low_level/start.jpg -filter -diff -merge -text
example_workflows/tricks/shrek2.jpg -filter -diff -merge -text
example_workflows/tricks/shrek3.jpg -filter -diff -merge -text
LICENSE
+346 -166
View File
@@ -1,201 +1,381 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
LTX-2 Community License Agreement
License date: January 5, 2026
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
By using or distributing any portion or element of LTX-2, you agree
to be bound by this Agreement.
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Agreement" means the terms and conditions for the license, use,
reproduction, and distribution of LTX-2 and the Complementary
Materials, as specified in this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Control" means the direct or indirect ownership of more than
fifty percent (50%) of the voting securities or other ownership
interests, or the power to direct the management and policies of
such Entity through voting rights, contract, or otherwise.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"Data" means a collection of information and/or content extracted
from the dataset used with LTX-2, including to train, pretrain,
or otherwise evaluate LTX-2. The Data is not licensed under this
Agreement.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Derivatives of LTX-2" means all modifications to LTX-2, works
based on LTX-2, or any other model which is created or initialized
by transfer of patterns of the weights, parameters, activations or
output of LTX-2, to the other model, in order to cause the other
model to perform similarly to LTX-2, including – but not limited
to - distillation methods entailing the use of intermediate data
representations or methods based on the generation of synthetic
data by LTX-2 for training the other model. For clarity, Derivatives
of LTX-2 include: (i) any fine-tuned or adapted weights, parameters,
or checkpoints derived from LTX-2; (ii) derivative model architectures
that incorporate or are based upon LTX-2's architecture; and
(iii) any modified or extended versions of the Complementary
Materials. All intellectual property rights in Derivatives of LTX-2
shall be subject to the terms of this Agreement, and you may not
claim exclusive ownership rights in any Derivatives of LTX-2 that
would restrict the rights granted herein.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Entity" means any individual, corporation, partnership, limited
liability company, or other legal entity. For purposes of this
Agreement, an Entity shall be deemed to include, on an aggregative
basis, all subsidiaries, affiliates, and other companies under
common Control with such Entity. When determining whether an Entity
meets any threshold under this Agreement (including revenue
thresholds), all subsidiaries, affiliates, and companies under
common Control shall be considered collectively.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Harm" includes but is not limited to physical, mental,
psychological, financial and reputational damage, pain, or loss.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Licensor" or "Lightricks" means the owner that is granting the
license under this Agreement. For the purposes of this Agreement,
the Licensor is Lightricks Ltd.
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"LTX-2" means the large language models, text/image/video/audio/3D
generation models, and multimodal large language models and their
software and algorithms, including trained model weights, parameters
(including optimizer states), machine-learning model code,
inference-enabling code, training-enabling code, fine-tuning
enabling code, accompanying source code, scripts, documentation,
tutorials, examples, and all other elements of the foregoing
distributed and made publicly available by Lightricks (including,
for example, at https://github.com/Lightricks/LTX-2) for the LTX-2
model released on January 5, 2026. This license is applicable to
all LTX-2 versions released since January 5, 2026, and all future
releases of LTX-2 under this license.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Output" means the results of operating LTX-2 as embodied in
informational content resulting therefrom.
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
"you" (or "your") means an individual or legal Entity licensing
LTX-2 in accordance with this Agreement and/or making use of LTX-2
for whichever purpose and in any field of use, including usage of
LTX-2 in an end-use application - e.g. chatbot, translator, image
generator.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
2. Grant of License. Subject to the terms and conditions of this
Agreement, you are granted a non-exclusive, worldwide,
non-transferable and royalty-free limited license under Licensor's
intellectual property or other rights owned by Licensor embodied
in LTX-2 to use, reproduce, prepare, distribute, publicly display,
publicly perform, sublicense, copy, create derivative works of,
and make modifications to LTX-2, for any purpose, subject to the
restrictions set forth in Attachment A; provided however, that
Entities with annual revenues of at least $10,000,000 (the
"Commercial Entities") are required to obtain a paid commercial
use license in order to use LTX-2 and Derivatives of LTX-2,
subject to the terms and provisions of a different license (the
"Commercial Use Agreement"), as will be provided by the Licensor.
Commercial Entities interested in such a commercial license are
required to contact Licensor. Any commercial use of LTX-2 or
Derivatives of LTX-2 by the Commercial Entities not in accordance
with this Agreement and/or the Commercial Use Agreement is strictly
prohibited and shall be deemed a material breach of this Agreement.
Such material breach will be subject, in addition to any license
fees owed to Licensor for the period such Commercial Entity used
LTX-2 (as will be determined by Licensor), to liquidated damages,
which will be paid to Licensor immediately upon demand, in an
amount equal to double the amount that would otherwise have been
paid by you for the relevant period of time. Such amount reflects
a reasonable estimation of the losses and administrative costs
incurred due to such breach. You agree and understand that this
remedy does not limit the Licensor's right to pursue other remedies
available at law or equity.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
3. Distribution and Redistribution. You may host for third parties
remote access purposes (e.g. software-as-a-service), reproduce
and distribute copies of LTX-2 or Derivatives of LTX-2 thereof in
any medium, with or without modifications, provided that you meet
the following conditions:
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) Use-based restrictions as referenced in paragraph 4 and all
provisions of Attachment A MUST be included as an enforceable
provision by you in any type of legal agreement (e.g. a
license) governing the use and/or distribution of LTX-2 or
Derivatives of LTX-2, and you shall give notice to subsequent
users you distribute to, that LTX-2 or Derivatives of LTX-2
are subject to paragraph 4 and Attachment A in their entirety,
including all use restrictions and acceptable use policies;
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must provide any third party recipients of LTX-2 or
Derivatives of LTX-2 a copy of this Agreement, including all
attachments and use policies. Any Derivative of LTX-2 (as
defined in Section 1, including but not limited to fine-tuned
weights, modified training code, models trained on Outputs, or
any other derivative) must be distributed exclusively under
the terms of this Agreement with a complete copy of this
license included;
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must cause any modified files to carry prominent notices
stating that you changed the files;
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) You must retain all copyright, patent, trademark, and
attribution notices excluding those notices that do not
pertain to any part of LTX-2, Derivatives of LTX-2.
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add your own copyright statement to your modifications and
may provide additional or different license terms and conditions -
respecting paragraph 3(a) - for use, reproduction, or distribution
of your modifications, or for any such Derivatives of LTX-2 as a
whole, provided your use, reproduction, and distribution of LTX-2
otherwise complies with the conditions stated in this Agreement,
and you provide a complete copy of this Agreement with any such
use, reproduction and distribution of LTX-2 and any Derivatives
thereof.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
4. Use-based restrictions. The restrictions set forth in Attachment A
are considered Use-based restrictions. Therefore, you cannot use
LTX-2 and the Derivatives of LTX-2 in violation of the specified
restricted uses. You may use LTX-2 subject to this Agreement,
including only for lawful purposes and in accordance with the
Agreement. "Use" may include creating any content with, fine-tuning,
updating, running, training, evaluating and/or re-parametrizing
LTX-2. You shall require all of your users who use LTX-2 or a
Derivative of LTX-2 to comply with the terms of this paragraph 4.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
5. The Output You Generate. Except as set forth herein, Licensor
claims no rights in the Output you generate using LTX-2. You are
accountable for input you insert into LTX-2, the Output you
generate and its subsequent uses. No use of the Output can
contravene any provision as stated in the Agreement.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
6. Updates and Runtime Restrictions. To the maximum extent permitted
by law, Licensor reserves the right to restrict (remotely or
otherwise) usage of LTX-2 in violation of this Agreement, update
LTX-2 through electronic means, or modify the Output of LTX-2
based on updates. You shall undertake reasonable efforts to use
the latest version of LTX-2. Any use of the non-current version
of LTX-2 is done solely at your risk.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
7. Export Controls and Sanctions Compliance. You acknowledge that
LTX-2, Derivatives of LTX-2 may be subject to export control laws
and regulations, including but not limited to the U.S. Export
Administration Regulations and sanctions programs administered by
the Office of Foreign Assets Control (OFAC). You represent and
warrant that you and any users of LTX-2 are not (i) located in,
organized under the laws of, or ordinarily resident in any country
or territory subject to comprehensive sanctions; (ii) identified
on any U.S. government restricted party list, including the
Specially Designated Nationals and Blocked Persons List; or
(iii) otherwise prohibited from receiving LTX-2 under applicable
law. You shall not export, re-export, or transfer LTX-2, directly
or indirectly, in violation of any applicable export control or
sanctions laws or regulations. You agree to comply with all
applicable trade control laws and shall indemnify and hold
Licensor harmless from any claims arising from your failure to
comply with such laws.
8. Trademarks and related. Nothing in this Agreement permits you to
make use of Licensor's trademarks, trade names, logos or to
otherwise suggest endorsement or misrepresent the relationship
between the parties; and any rights not expressly granted herein
are reserved by the Licensor.
9. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides LTX-2 on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
implied, including, without limitation, any warranties or
conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS
FOR A PARTICULAR PURPOSE. You are solely responsible for
determining the appropriateness of using or redistributing LTX-2
and Derivatives of LTX-2 and assume any risks associated with
your exercise of permissions under this Agreement.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
10. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall Licensor be liable
to you for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as
a result of this Agreement or out of the use or inability to use
LTX-2 (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if Licensor has been
advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
11. Accepting Warranty or Additional Liability. While redistributing
LTX-2 and Derivatives of LTX-2, you may, provided you do not
violate the terms of this Agreement, choose to offer and charge
a fee for, acceptance of support, warranty, indemnity, or other
liability obligations. However, in accepting such obligations,
you may act only on your own behalf and on your sole
responsibility, not on behalf of Licensor, and only if you agree
to indemnify, defend, and hold Licensor harmless for any liability
incurred by, or claims asserted against Licensor, by reason of
your accepting any such warranty or additional liability.
12. Governing Law. This Agreement and all relations, disputes, claims
and other matters arising hereunder (including non-contractual
disputes or claims) will be governed exclusively by, and construed
exclusively in accordance with, the laws of the State of New York.
To the extent permitted by law, choice of laws rules and the
United Nations Convention on Contracts for the International Sale
of Goods will not apply. For the purposes of adjudicating any
action or proceeding to enforce the terms of this Agreement, you
hereby irrevocably consent to the exclusive jurisdiction of, and
venue in, the federal and state courts located in the County of
New York within the State of New York. The prevailing party in
any claim or dispute between the parties under this Agreement
will be entitled to reimbursement of its reasonable attorneys'
fees and costs. You hereby waive the right to a trial by jury,
to participate in a class or representative action (including in
arbitration), or to combine individual proceedings in court or
in arbitration without the consent of all parties.
13. Term and Termination. This Agreement is effective upon your
acceptance and continues until terminated. Licensor may terminate
this Agreement immediately upon written notice to you if you
breach any provision of this Agreement, including but not limited
to violations of the use restrictions in Attachment A or
unauthorized commercial use. Upon termination: (a) all rights
granted to you under this Agreement will immediately cease;
(b) you must immediately cease all use of LTX-2 and Derivatives
of LTX-2; (c) you must delete or destroy all copies of LTX-2
and Derivatives of LTX-2 in your possession or control; and
(d) you must notify any third parties to whom you distributed
LTX-2 or Derivatives of LTX-2 of the termination. Sections 8-13,
and Section 15 shall survive termination of this Agreement.
Termination does not relieve you of any obligations incurred
prior to termination, including payment obligations under
Section 2. In addition, if You commence a lawsuit or other
proceedings (including a cross-claim or counterclaim in a lawsuit)
against Licensor or any person or entity alleging that LTX-2 or
any Output, or any portion of any of the foregoing, infringe any
intellectual property or other right owned or licensable by you,
then all licenses granted to you under this Agreement shall
terminate as of the date such lawsuit or other proceeding is filed.
14. Disputes and Arbitration. All disputes arising in connection with
this Agreement shall be finally settled by arbitration under the
Rules of Arbitration of the International Chamber of Commerce
("ICC Rules"), by one (1) arbitrator appointed in accordance with
the ICC Rules. The seat of arbitration shall be New York, NY, USA,
and the proceedings shall be conducted in English. The arbitrator
shall be empowered to grant any relief that a court could grant.
Judgment on the arbitration award may be entered by any court
having jurisdiction thereof. Each party waives its right to a
trial by jury and to participate in any class or representative
action.
15. If any provision of this Agreement is held to be
invalid, illegal
or unenforceable, the remaining provisions shall be unaffected
thereby and remain valid as if such provision had not been set
forth herein.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
ATTACHMENT A: Use Restrictions
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
When using the Outputs, LTX-2 and any Derivatives thereof, you
will comply with the Acceptable Use Policy. In addition, you
agree not to use the Outputs, LTX-2 or its Derivatives in any
of the following ways:
Copyright [yyyy] [name of copyright owner]
1. In any way that violates any applicable national, federal,
state, local or international law or regulation;
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
2. For the purpose of exploiting, Harming or attempting to
exploit or Harm minors in any way;
http://www.apache.org/licenses/LICENSE-2.0
3. To generate or disseminate false information and/or content
with the purpose of Harming others;
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
4. To generate or disseminate personal identifiable information
that can be used to Harm an individual;
5. To generate or disseminate information and/or content (e.g.
images, code, posts, articles), and place the information
and/or content in any context (e.g. bot generating tweets)
without expressly and intelligibly disclaiming that the
information and/or content is machine generated;
6. To defame, disparage or otherwise harass others;
7. To impersonate or attempt to impersonate (e.g. deepfakes)
others without their consent;
8. For fully automated decision making that adversely impacts an
individual's legal rights or otherwise creates or modifies a
binding, enforceable obligation;
9. For any use intended to or which has the effect of
discriminating against or Harming individuals or groups based
on online or offline social behavior or known or predicted
personal or personality characteristics;
10. To exploit any of the vulnerabilities of a specific group of
persons based on their age, social, physical or mental
characteristics, in order to materially distort the behavior
of a person pertaining to that group in a manner that causes
or is likely to cause that person or another person physical
or psychological Harm;
11. For any use intended to or which has the effect of
discriminating against individuals or groups based on legally
protected characteristics or categories;
12. To provide medical advice and medical results interpretation;
13. To generate or disseminate information for the purpose to be
used for administration of justice, law enforcement,
immigration or asylum processes, such as predicting an
individual will commit fraud/crime commitment (e.g. by text
profiling, drawing causal relationships between assertions
made in documents, indiscriminate and arbitrarily-targeted use);
14. To generate and/or disseminate malware (including – but not
limited to – ransomware) or any other content to be used for
the purpose of harming electronic systems;
15. To engage in, promote, incite, or facilitate discrimination
or other unlawful or harmful conduct in the provision of
employment, employment benefits, credit, housing, or other
essential goods and services;
16. To engage in, promote, incite, or facilitate the harassment,
abuse, threatening, or bullying of individuals or groups of
individuals;
17. For military, warfare, nuclear industries or applications,
weapons development, or any use in connection with activities
that may cause death, personal injury, or severe physical or
environmental damage;
18. For commercial use only: To train, improve, or fine-tune any
other machine learning model, artificial intelligence system,
or competing model, except for Derivatives of LTX-2 as
expressly permitted under this Agreement;
19. To circumvent, disable, or interfere with any technical
limitations, safety features, content filters, or use
restrictions implemented in LTX-2 by Licensor;
20. To use LTX-2 or Derivatives of LTX-2 in any product, service,
or application that directly competes with Licensor's
commercial products or services, or is designed to replace or
substitute Licensor's offerings in the market, without
obtaining a separate commercial license from Licensor.
README.md
+75 -140
View File
@@ -1,162 +1,97 @@
# ComfyUI-LTXVideo
[![GitHub](https://img.shields.io/badge/LTXV-Repo-blue?logo=github)](https://github.com/Lightricks/LTX-Video)
[![Website](https://img.shields.io/badge/Website-LTXV-181717?logo=google-chrome)](https://www.lightricks.com/ltxv)
[![Model](https://img.shields.io/badge/HuggingFace-Model-orange?logo=huggingface)](https://huggingface.co/Lightricks/LTX-Video)
[![LTXV Trainer](https://img.shields.io/badge/LTXV-Trainer%20Repo-9146FF)](https://github.com/Lightricks/LTX-Video-Trainer)
[![Demo](https://img.shields.io/badge/Demo-Try%20Now-brightgreen?logo=vercel)](https://app.ltx.studio/ltx-video)
[![Paper](https://img.shields.io/badge/Paper-arXiv-B31B1B?logo=arxiv)](https://arxiv.org/abs/2501.00103)
[![Discord](https://img.shields.io/badge/Join-Discord-5865F2?logo=discord)](https://discord.gg/Mn8BRgUKKy)
ComfyUI-LTXVideo is a collection of custom nodes for ComfyUI, designed to provide useful tools for working with the LTXV model.
The model itself is supported in the core ComfyUI [code](https://github.com/comfyanonymous/ComfyUI/tree/master/comfy/ldm/lightricks).
The main LTXVideo repository can be found [here](https://github.com/Lightricks/LTX-Video).
## 🚀 New to using LTXV with ComfyUI? See our [Getting Started](https://github.com/Lightricks/ComfyUI-LTXVideo/blob/master/getting_started.md) page
[![GitHub](https://img.shields.io/badge/LTX-Repo-blue?logo=github)](https://github.com/Lightricks/LTX-2)
[![Website](https://img.shields.io/badge/Website-LTX-181717?logo=google-chrome)](https://ltx.io/model)
[![Model](https://img.shields.io/badge/HuggingFace-Model-orange?logo=huggingface)](https://huggingface.co/Lightricks/LTX-2)
[![LTXV Trainer](https://img.shields.io/badge/LTX-Trainer%20Repo-9146FF)](https://github.com/Lightricks/LTX-2/tree/main/packages/ltx-trainer)
[![Demo](https://img.shields.io/badge/Demo-Try%20Now-brightgreen?logo=vercel)](https://app.ltx.studio/ltx-2-playground/i2v)
[![Paper](https://img.shields.io/badge/Paper-arXiv-B31B1B?logo=arxiv)](https://videos.ltx.io/LTX-2/grants/LTX_2_Technical_Report_compressed.pdf)
[![Discord](https://img.shields.io/badge/Join-Discord-5865F2?logo=discord)](https://discord.gg/ltxplatform)
# ⭐ 16.07.2025 - LTXV 0.9.8 Release ⭐
A collection of powerful custom nodes that extend ComfyUI's capabilities for the LTX-2 video generation model.
### 🚀 What's New
1. **LTXV 0.9.8 Model**<br>
The new model and its distilled variants offer improved prompt understanding and detail generation<br>
👉 [13B Distilled model](https://huggingface.co/Lightricks/LTX-Video/blob/main/ltxv-13b-0.9.8-distilled.safetensors)<br>
👉 [13B Distilled model 8-bit](https://huggingface.co/Lightricks/LTX-Video/blob/main/ltxv-13b-0.9.8-distilled-fp8.safetensors)<br>
👉 [2B from 13B Distilled model](https://huggingface.co/Lightricks/LTX-Video/blob/main/ltxv-2b-0.9.8-distilled.safetensors)<br>
👉 [2B from 13B Distilled model 8-bit](https://huggingface.co/Lightricks/LTX-Video/blob/main/ltxv-2b-0.9.8-distilled-fp8.safetensors)<br>
👉 [IC Lora Detailer](https://huggingface.co/Lightricks/LTX-Video-ICLoRA-detailer-13b-0.9.8/resolve/main/ltxv-098-ic-lora-detailer-comfyui.safetensors)<br>
LTX-2 is built into ComfyUI core ([see it here](https://github.com/comfyanonymous/ComfyUI/tree/master/comfy/ldm/lightricks)), making it readily accessible to all ComfyUI users. This repository hosts additional nodes and workflows to help you get the most out of LTX-2's advanced features.
**To learn more about LTX-2** See the [main LTX-2 repository](https://github.com/Lightricks/LTX-2) for model details and additional resources.
2. **Autoregressive Generation**
Introducing new ComfyUI nodes that enable virtually infinite video generation. The new **LTXV Looping Sampler** node allows generation of videos with arbitrary length and consistent motion. **ICLoRAs** are supported as well—by providing guidance from existing videos (e.g., depth, pose, or Canny edges), you can generate long videos in a video-to-video manner.
👉 [Long Img2Video Generation Flow](#long-video-generation)
👉 [Long Video2Video Generation Flow](#long-video-generation)
## Prerequisites
Before you begin using an LTX-2 workflow in ComfyUI, make sure you have:
3. **Detailer ICLoRA**
Introducing the **Detailer ICLoRA**, which enhances generated latents with fine details by applying a few additional diffusion steps. This results in significantly more detailed generations.
👉 [Detailer ICLoRA Flow](#video-upscaling)
# ⭐ 8.07.2025 - LTXVideo ICLora Release ⭐
### 🚀 What's New in LTXVideo ICLoRA
1. **Three New ICLoRA Models**
Introducing powerful in-context LoRA models that enable precise control over video generation:
- Depth Control: [LTX-Video-ICLoRA-depth-13b-0.9.7](https://huggingface.co/Lightricks/LTX-Video-ICLoRA-depth-13b-0.9.7)
- Pose Control: [LTX-Video-ICLoRA-pose-13b-0.9.7](https://huggingface.co/Lightricks/LTX-Video-ICLoRA-pose-13b-0.9.7)
- Edge/Canny Control: [LTX-Video-ICLoRA-canny-13b-0.9.7](https://huggingface.co/Lightricks/LTX-Video-ICLoRA-canny-13b-0.9.7)
2. **New Node: 🅛🅣🅧 LTXV In Context Sampler**
A dedicated node for seamlessly integrating ICLoRA models into your workflow, enabling fine-grained control over video generation using depth maps, pose estimation, or edge detection.
3. **Example Workflow**
Check out [example workflow](#iclora) for a complete example demonstrating how to use the ICLoRA models effectively.
4. **Custom ICLoRA Training**
We've released a trainer that allows you to create your own specialized ICLoRA models for custom control signals. Check out the [trainer repository](https://github.com/Lightricks/LTX-Video-Trainer) to get started.
* ComfyUI installed (Download here](https://www.comfy.org/download)
* CUDA-compatible GPU with 32GB+ VRAM
* 100GB+ free disk space for models and cache
# ⭐ 9.06.2025 – LTXVideo VAE Patcher, Mask manipulation and Q8 LoRA loader nodes. ⭐
1. **LTXV Patcher VAE**<br> The new node improves VAE decoding performance by reducing runtime and cutting memory consumption by up to 50%. This allows generation of higher-resolution outputs on consumer-grade GPUs with limited VRAM, without needing to load the VAE partially or decode in tiles.<br>
⚠️ On *Windows*, you may need to add the paths to the *MSVC compiler (cl.exe)* and *ninja.exe* to your system environment PATH variable. <br>
2. **LTXV Preprocess Masks**<br>
Preprocesses masks for use with the LTXVideo model's latent masking. It validates mask dimensions based on VAE downscaling, supports optional inversion, handles the first frame mask separately, combines temporal masks via max pooling, applies morphological operations to grow or shrink masks, and clamps values to ensure correct opacity. The result is a set of masks ready for latent-space masking.
3. **LTXV Q8 Lora Model Loader**<br>
Applying LoRA to an FP8-quantized model requires special handling to preserve output quality. It's crucial to apply LoRA weights using the correct precision, as the current LoRA implementation in ComfyUI does so in a non-optimal way. This node addresses that limitation by ensuring LoRA weights are applied properly, resulting in significantly better quality. If you're working with an FP8 LTXV model, using this node guarantees that LoRA behaves as expected and delivers the intended effect.
## Quick Start 🚀
We recommend using the LTX-2 workflows available in Comfy Manager.
1. Open ComfyUI
2. Click the Manager button (or press Ctrl+M)
3. Select Install Custom Nodes
4. Search for “LTXVideo”
5. Click Install
6. Wait for installation to complete
7. Restart ComfyUI
The nodes will appear in your node menu under the “LTXVideo” category. Required models will be downloaded on first use.
# ⭐ 14.05.2025 – LTXVideo 13B 0.9.7 Distilled Release ⭐
## Example Workflows
### 🚀 What's New in LTXVideo 13B 0.9.7 Distilled
1. **LTXV 13B Distilled 🥳 0.9.7**<br>
Delivers cinematic-quality videos at fraction of steps needed to run full model. Only 4 or 8 steps needed for single generation.<br>
👉 [Download here](https://huggingface.co/Lightricks/LTX-Video/blob/main/ltxv-13b-0.9.7-distilled.safetensors)
The ComfyUI-LTXVideo installation includes several example workflows.
You can see them all at:
'''
ComfyUI/custom_nodes/ComfyUI-LTXVideo/example_workflows/
'''
2. **LTXV 13B Distilled Quantized 0.9.7**<br>
Offers reduced memory requirements and even faster inference speeds.
Ideal for consumer-grade GPUs (e.g., NVIDIA 4090, 5090).<br>
***Important:*** In order to get the best performance with the quantized version please install [q8_kernels](https://github.com/Lightricks/LTXVideo-Q8-Kernels) package and use dedicated flow below. <br>
👉 [Download here](https://huggingface.co/Lightricks/LTX-Video/blob/main/ltxv-13b-0.9.7-distilled-fp8.safetensors)<br>
🧩 Example ComfyUI flow available in the [Example Workflows](#example-workflows) section.
* [`Text to video full model`](./example_workflows/LTX-2_T2V_Full_wLora.json)
* [`Text to video distilled model (Fast)`](./example_workflows/LTX-2_T2V_Distilled_wLora.json)
* [`Image to video full model`](./example_workflows/LTX-2_I2V_Full_wLora.json)
* [`Image to video distilled model (Fast)`](./example_workflows/LTX-2_I2V_Distilled_wLora.json)
* [`Video to video detailer`](./example_workflows/LTX-2_V2V_Detailer.json)
* [`IC-LoRA distilled model (depth + human pose + edges)`](./example_workflows/LTX-2_ICLoRA_All_Distilled.json)
3. **Updated LTV 13B Quantized version**<br>
From now on all our 8bit quantized models are running natively in ComfyUI, still with our Q8 patcher node you will get the best inference speed.<br>
👉 [Download here](https://huggingface.co/Lightricks/LTX-Video/blob/main/ltxv-13b-0.9.7-dev-fp8.safetensors)<br>
# ⭐ 06.05.2025 – LTXVideo 13B 0.9.7 Release ⭐
## Required Models
### 🚀 What's New in LTXVideo 13B 0.9.7
Download the following models:
1. **LTXV 13B 0.9.7**
Delivers cinematic-quality videos at unprecedented speed.<br>
👉 [Download here](https://huggingface.co/Lightricks/LTX-Video/blob/main/ltxv-13b-0.9.7-dev.safetensors)
**LTX-2 Model Checkpoint** (choose and download one of the following)
* [`ltx-2-19b-dev-fp8.safetensors`](https://huggingface.co/Lightricks/LTX-2/blob/main/ltx-2-19b-dev-fp8.safetensors)
* [`ltx-2-19b-distilled-fp8.safetensors`](https://huggingface.co/Lightricks/LTX-2/blob/main/ltx-2-19b-distilled-fp8.safetensors)
* [`ltx-2-19b-dev.safetensors`](https://huggingface.co/Lightricks/LTX-2/blob/main/ltx-2-19b-dev.safetensors)
* [`ltx-2-19b-distilled.safetensors`](https://huggingface.co/Lightricks/LTX-2/blob/main/ltx-2-19b-distilled.safetensors)
2. **LTXV 13B Quantized 0.9.7**
Offers reduced memory requirements and even faster inference speeds.
Ideal for consumer-grade GPUs (e.g., NVIDIA 4090, 5090).
Delivers outstanding quality with improved performance.<br>
***Important:*** In order to run the quantized version please install [LTXVideo-Q8-Kernels](https://github.com/Lightricks/LTXVideo-Q8-Kernels) package and use dedicated flow below. Loading the model in Comfy with LoadCheckpoint node won't work. <br>
👉 [Download here](https://huggingface.co/Lightricks/LTX-Video/blob/main/ltxv-13b-0.9.7-dev-fp8.safetensors)<br>
🧩 Example ComfyUI flow available in the [Example Workflows](#example-workflows) section.
**Spatial Upsampler** - Required for current two-stage pipeline implementations in this repository
* [`ltx-2-spatial-upsampler-x2-1.0.safetensors`](https://huggingface.co/Lightricks/LTX-2/blob/main/ltx-2-spatial-upsampler-x2-1.0.safetensors)
3. **Latent Upscaling Models**
Enables inference across multiple scales by upscaling latent tensors without decoding/encoding.
Multiscale inference delivers high-quality results in a fraction of the time compared to similar models.<br>
***Important:*** Make sure you put the models below in **models/upscale_models** folder.<br>
👉 Spatial upscaling: [Download here](https://huggingface.co/Lightricks/LTX-Video/blob/main/ltxv-spatial-upscaler-0.9.7.safetensors).<br>
👉 Temporal upscaling: [Download here](https://huggingface.co/Lightricks/LTX-Video/blob/main/ltxv-temporal-upscaler-0.9.7.safetensors).<br>
🧩 Example ComfyUI flow available in the [Example Workflows](#example-workflows) section.
**Temporal Upsampler** - Required for current two-stage pipeline implementations in this repository
* [`ltx-2-temporal-upsampler-x2-1.0.safetensors`](https://huggingface.co/Lightricks/LTX-2/blob/main/ltx-2-temporal-upsampler-x2-1.0.safetensors)
**Distilled LoRA** - Required for current two-stage pipeline implementations in this repository (except DistilledPipeline and ICLoraPipeline)
* [`ltx-2-19b-distilled-lora-384.safetensors`](https://huggingface.co/Lightricks/LTX-2/blob/main/ltx-2-19b-distilled-lora-384.safetensors)
**Gemma Text Encoder** (download all assets from the repository)
* [`Gemma 3`](https://huggingface.co/google/gemma-3-12b-it-qat-q4_0-unquantized/blob/main)
**LoRAs**
* [`ltx-2-19b-ic-lora-canny-control.safetensors`](https://huggingface.co/Lightricks/LTX-2-19b-IC-LoRA-Canny-Control/blob/main/ltx-2-19b-ic-lora-canny-control.safetensors)
* [`ltx-2-19b-ic-lora-depth-control.safetensors`](https://huggingface.co/Lightricks/LTX-2-19b-IC-LoRA-Depth-Control/blob/main/ltx-2-19b-ic-lora-depth-control.safetensors)
* [`ltx-2-19b-ic-lora-detailer.safetensors`](https://huggingface.co/Lightricks/LTX-2-19b-IC-LoRA-Detailer/blob/main/ltx-2-19b-ic-lora-detailer.safetensors)
* [`ltx-2-19b-ic-lora-pose-control.safetensors`](https://huggingface.co/Lightricks/LTX-2-19b-IC-LoRA-Pose-Control/blob/main/ltx-2-19b-ic-lora-pose-control.safetensors)
* [`ltx-2-19b-lora-camera-control-dolly-in.safetensors`](https://huggingface.co/Lightricks/LTX-2-19b-LoRA-Camera-Control-Dolly-In/blob/main/ltx-2-19b-lora-camera-control-dolly-in.safetensors)
* [`ltx-2-19b-lora-camera-control-dolly-left.safetensors`](https://huggingface.co/Lightricks/LTX-2-19b-LoRA-Camera-Control-Dolly-Left/blob/main/ltx-2-19b-lora-camera-control-dolly-left.safetensors)
* [`ltx-2-19b-lora-camera-control-dolly-out.safetensors`](https://huggingface.co/Lightricks/LTX-2-19b-LoRA-Camera-Control-Dolly-Out/blob/main/ltx-2-19b-lora-camera-control-dolly-out.safetensors)
* [`ltx-2-19b-lora-camera-control-dolly-right.safetensors`](https://huggingface.co/Lightricks/LTX-2-19b-LoRA-Camera-Control-Dolly-Right/blob/main/ltx-2-19b-lora-camera-control-dolly-right.safetensors)
* [`ltx-2-19b-lora-camera-control-jib-down.safetensors`](https://huggingface.co/Lightricks/LTX-2-19b-LoRA-Camera-Control-Jib-Down/blob/main/ltx-2-19b-lora-camera-control-jib-down.safetensors)
* [`ltx-2-19b-lora-camera-control-jib-up.safetensors`](https://huggingface.co/Lightricks/LTX-2-19b-LoRA-Camera-Control-Jib-Up/blob/main/ltx-2-19b-lora-camera-control-jib-up.safetensors)
* [`ltx-2-19b-lora-camera-control-static.safetensors`](https://huggingface.co/Lightricks/LTX-2-19b-LoRA-Camera-Control-Static/blob/main/ltx-2-19b-lora-camera-control-static.safetensors)
### Technical Updates
1. ***New simplified flows and nodes***<br>
1.1. Simplified image to video: [Download here](example_workflows/ltxv-13b-i2v-base.json).<br>
1.2. Simplified image to video with extension: [Download here](example_workflows/ltxv-13b-i2v-extend.json).<br>
1.3. Simplified image to video with keyframes: [Download here](example_workflows/ltxv-13b-i2v-keyframes.json).<br>
# 17.04.2025 ⭐ LTXVideo 0.9.6 Release ⭐
### LTXVideo 0.9.6 introduces:
1. LTXV 0.9.6 – higher quality, faster, great for final output. Download from [here](https://huggingface.co/Lightricks/LTX-Video/resolve/main/ltxv-2b-0.9.6-dev-04-25.safetensors).
2. LTXV 0.9.6 Distilled – our fastest model yet (only 8 steps for generation), lighter, great for rapid iteration. Download from [here](https://huggingface.co/Lightricks/LTX-Video/resolve/main/ltxv-2b-0.9.6-distilled-04-25.safetensors).
### Technical Updates
We introduce the __STGGuiderAdvanced__ node, which applies different CFG and STG parameters at various diffusion steps. All flows have been updated to use this node and are designed to provide optimal parameters for the best quality.
See the [Example Workflows](#example-workflows) section.
# 5.03.2025 ⭐ LTXVideo 0.9.5 Release ⭐
### LTXVideo 0.9.5 introduces:
1. Improved quality with reduced artifacts.
2. Support for higher resolution and longer sequences.
3. Frame and sequence conditioning (beyond the first frame).
4. Enhanced prompt understanding.
5. Commercial license availability.
### Technical Updates
Since LTXVideo is now fully supported in the ComfyUI core, we have removed the custom model implementation. Instead, we provide updated workflows to showcase the new features:
1. **Frame Conditioning** – Enables interpolation between given frames.
2. **Sequence Conditioning** – Allows motion interpolation from a given frame sequence, enabling video extension from the beginning, end, or middle of the original video.
3. **Prompt Enhancer** – A new node that helps generate prompts optimized for the best model performance.
See the [Example Workflows](#example-workflows) section for more details.
### LTXTricks Update
The LTXTricks code has been integrated into this repository (in the `/tricks` folder) and will be maintained here. The original [repo](https://github.com/logtd/ComfyUI-LTXTricks) is no longer maintained, but all existing workflows should continue to function as expected.
## 22.12.2024
Fixed a bug which caused the model to produce artifacts on short negative prompts when using a native CLIP Loader node.
## 19.12.2024 ⭐ Update ⭐
1. Improved model - removes "strobing texture" artifacts and generates better motion. Download from [here](https://huggingface.co/Lightricks/LTX-Video/resolve/main/ltx-video-2b-v0.9.1.safetensors).
2. STG support
3. Integrated image degradation system for improved motion generation.
4. Additional initial latent optional input to chain latents for high res generation.
5. Image captioning in image to video [flow](example_workflows/ltxvideo-i2v.json).
## Advanced Techniques
### Low VRAM
* For systems with low VRAM you can use the model loader nodes from [low_vram_loaders.py](./low_vram_loaders.py). Those nodes ensure the correct order of execution and perform the model offloading such that generation fits in 32 GB VRAM.
* For complete information about using LTX-2 models, workflows, and nodes in ComfyUI, please visit our [Open Source documentation](https://docs.ltx.video/open-source-model/integration-tools/comfy-ui).
__init__.py
+36 -7
View File
@@ -1,16 +1,33 @@
from .decoder_noise import DecoderNoise
from .dynamic_conditioning import DynamicConditioning
from .easy_samplers import (
LinearOverlapLatentTransition,
LTXVBaseSampler,
LTXVExtendSampler,
LTXVInContextSampler,
)
from .film_grain import LTXVFilmGrain
from .gemma_encoder import LTXVGemmaCLIPModelLoader, LTXVGemmaEnhancePrompt
from .gemma_encoder_mgpu_loader import LTXVGemmaCLIPModelLoaderMGPU
from .guide import LTXVAddGuideAdvanced
from .latent_adain import LTXVAdainLatent
from .latent_upsampler import LTXVLatentUpsampler
from .latents import LTXVSelectLatents, LTXVSetVideoLatentNoiseMasks
from .guiders import GuiderParametersNode, MultimodalGuiderNode
from .latent_norm import (
LTXVAdainLatent,
LTXVPerStepAdainPatcher,
LTXVPerStepStatNormPatcher,
LTXVStatNormLatent,
)
from .latents import (
LTXVAddLatentGuide,
LTXVImgToVideoConditionOnly,
LTXVSelectLatents,
LTXVSetVideoLatentNoiseMasks,
)
from .looping_sampler import LTXVLoopingSampler, MultiPromptProvider
from .low_vram_loaders import (
LowVRAMAudioVAELoader,
LowVRAMCheckpointLoader,
LowVRAMLatentUpscaleModelLoader,
)
from .masks import LTXVPreprocessMasks
from .nodes_registry import NODE_CLASS_MAPPINGS as RUNTIME_NODE_CLASS_MAPPINGS
from .nodes_registry import (
@@ -30,21 +47,22 @@ from .tiled_vae_decode import LTXVTiledVAEDecode
from .tricks import NODE_CLASS_MAPPINGS as TRICKS_NODE_CLASS_MAPPINGS
from .tricks import NODE_DISPLAY_NAME_MAPPINGS as TRICKS_NODE_DISPLAY_NAME_MAPPINGS
from .utiltily_nodes import ImageToCPU
from .vae_patcher.vae_patcher import LTXVPatcherVAE
from .vae_patcher import LTXVPatcherVAE
# Static node mappings, required for ComfyUI-Manager mapping to work
NODE_CLASS_MAPPINGS = {
"Set VAE Decoder Noise": DecoderNoise,
"LTXVLinearOverlapLatentTransition": LinearOverlapLatentTransition,
"LTXVAddGuideAdvanced": LTXVAddGuideAdvanced,
"LTXVAddLatentGuide": LTXVAddLatentGuide,
"LTXVAdainLatent": LTXVAdainLatent,
"LTXVImgToVideoConditionOnly": LTXVImgToVideoConditionOnly,
"LTXVPerStepAdainPatcher": LTXVPerStepAdainPatcher,
"LTXVApplySTG": LTXVApplySTG,
"LTXVBaseSampler": LTXVBaseSampler,
"LTXVInContextSampler": LTXVInContextSampler,
"LTXVExtendSampler": LTXVExtendSampler,
"LTXVFilmGrain": LTXVFilmGrain,
"LTXVPreprocessMasks": LTXVPreprocessMasks,
"LTXVLatentUpsampler": LTXVLatentUpsampler,
"LTXVPatcherVAE": LTXVPatcherVAE,
"LTXVPromptEnhancer": LTXVPromptEnhancer,
"LTXVPromptEnhancerLoader": LTXVPromptEnhancerLoader,
@@ -55,11 +73,22 @@ NODE_CLASS_MAPPINGS = {
"LTXVTiledSampler": LTXVTiledSampler,
"LTXVLoopingSampler": LTXVLoopingSampler,
"LTXVTiledVAEDecode": LTXVTiledVAEDecode,
"MultimodalGuider": MultimodalGuiderNode,
"GuiderParameters": GuiderParametersNode,
"STGAdvancedPresets": STGAdvancedPresetsNode,
"STGGuiderAdvanced": STGGuiderAdvancedNode,
"STGGuiderNode": STGGuiderNode,
"LTXVMultiPromptProvider": MultiPromptProvider,
"ImageToCPU": ImageToCPU,
"LTXVStatNormLatent": LTXVStatNormLatent,
"LTXVPerStepStatNormPatcher": LTXVPerStepStatNormPatcher,
"LTXVGemmaCLIPModelLoader": LTXVGemmaCLIPModelLoader,
"LTXVGemmaCLIPModelLoaderMGPU": LTXVGemmaCLIPModelLoaderMGPU,
"LTXVGemmaEnhancePrompt": LTXVGemmaEnhancePrompt,
"DynamicConditioning": DynamicConditioning,
"LowVRAMCheckpointLoader": LowVRAMCheckpointLoader,
"LowVRAMAudioVAELoader": LowVRAMAudioVAELoader,
"LowVRAMLatentUpscaleModelLoader": LowVRAMLatentUpscaleModelLoader,
}
# Consistent display names between static and dynamic node mappings in nodes_registry.py,
dynamic_conditioning.py
+57
View File
@@ -0,0 +1,57 @@
import torch
from .nodes_registry import comfy_node
@comfy_node(name="DynamicConditioning")
class DynamicConditioning:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"model": ("MODEL",),
"power": ("FLOAT", {"default": 1.3, "min": 1, "max": 2, "step": 0.01}),
"only_first_frame": ("BOOLEAN", {"default": True}),
}
}
RETURN_TYPES = ("MODEL",)
FUNCTION = "apply"
CATEGORY = "lightricks/LTXV"
INIT = False
def apply(self, model, power, only_first_frame):
self.only_first_frame = only_first_frame
self.power = power
model = model.clone()
model.set_model_denoise_mask_function(self.forward)
return (model,)
def find_step(self, sigma: torch.Tensor, step_sigmas: torch.Tensor):
for i, step_sigma in enumerate(step_sigmas):
if step_sigma <= sigma:
return i
return len(step_sigmas) - 1
def forward(
self, sigma: torch.Tensor, denoise_mask: torch.Tensor, extra_options: dict
):
model = extra_options["model"]
step_sigmas = extra_options["sigmas"]
step = self.find_step(sigma, step_sigmas)
# In order to apply power multiple times, this is the same as applying power number of times equal to step
power = self.power**step
denoise_mask = denoise_mask.clone()
if self.only_first_frame:
num_channels = model.model_patcher.model.diffusion_model.in_channels
denoise_mask[:, :num_channels, :1] **= power
else:
denoise_mask **= power
# make sure to update the denoise mask in the model, to get correct timestep values for all tokens
for k in model.conds:
if "positive" in k or "negative" in k:
for cond in model.conds[k]:
if "model_conds" in cond and "denoise_mask" in cond["model_conds"]:
cond["model_conds"]["denoise_mask"].cond = denoise_mask
# print(f"DynamicConditioning: power: {power}, step: {step}, sigma: {sigma}, step_sigmas: {step_sigmas}")
return denoise_mask
easy_samplers.py
+313 -104
View File
@@ -4,16 +4,30 @@ import comfy
import comfy_extras
import nodes
import torch
from comfy_extras.nodes_custom_sampler import SamplerCustomAdvanced
from comfy_extras.nodes_custom_sampler import SamplerCustomAdvanced, SplitSigmas
from comfy_extras.nodes_lt import EmptyLTXVLatentVideo, LTXVAddGuide, LTXVCropGuides
from nodes import VAEEncode
from .guide import blur_internal
from .latent_adain import LTXVAdainLatent
from .latent_norm import LTXVAdainLatent
from .latents import LTXVAddLatentGuide, LTXVSelectLatents
from .nodes_registry import comfy_node
def _get_raw_conds_from_guider(guider):
if not hasattr(guider, "raw_conds"):
if "negative" not in guider.original_conds:
# for BasicGuider
raise ValueError(
"Guider does not have negative conds, cannot use it as a guider."
)
raw_pos = guider.original_conds["positive"]
positive = [[raw_pos[0]["cross_attn"], copy.deepcopy(raw_pos[0])]]
raw_neg = guider.original_conds["negative"]
negative = [[raw_neg[0]["cross_attn"], copy.deepcopy(raw_neg[0])]]
guider.raw_conds = (positive, negative)
return guider.raw_conds
@comfy_node(
name="LTXVBaseSampler",
)
@@ -104,7 +118,7 @@ class LTXVBaseSampler:
}
RETURN_TYPES = ("LATENT", "CONDITIONING", "CONDITIONING")
RETURN_NAMES = ("denoised_output", "positive", "negative")
RETURN_NAMES = ("denoised", "positive", "negative")
FUNCTION = "sample"
CATEGORY = "sampling"
@@ -129,7 +143,12 @@ class LTXVBaseSampler:
optional_negative_index=-1,
optional_negative_index_strength=1.0,
optional_initialization_latents=None,
guiding_start_step=0,
guiding_end_step=1000,
):
guider = copy.copy(guider)
guider.original_conds = copy.deepcopy(guider.original_conds)
positive, negative = _get_raw_conds_from_guider(guider)
if optional_cond_images is not None:
optional_cond_images = (
@@ -143,7 +162,7 @@ class LTXVBaseSampler:
.movedim(1, -1)
.clamp(0, 1)
)
optional_cond_images = comfy_extras.nodes_lt.LTXVPreprocess().preprocess(
optional_cond_images = comfy_extras.nodes_lt.LTXVPreprocess.execute(
optional_cond_images, crf
)[0]
for i in range(optional_cond_images.shape[0]):
@@ -158,52 +177,60 @@ class LTXVBaseSampler:
optional_cond_images
), "Number of optional cond images must match number of optional cond indices"
try:
positive, negative = guider.raw_conds
except AttributeError:
raise ValueError(
"Guider does not have raw conds, cannot use it as a guider. "
"Please use STGGuiderAdvanced."
)
if optional_initialization_latents is None:
(latents,) = EmptyLTXVLatentVideo().generate(width, height, num_frames, 1)
(latents,) = EmptyLTXVLatentVideo().execute(width, height, num_frames, 1)
else:
latents = optional_initialization_latents
if (
optional_cond_images is not None
and optional_cond_images.shape[0] == 1
and optional_cond_indices[0] == 0
):
pixels = comfy.utils.common_upscale(
optional_cond_images[0].unsqueeze(0).movedim(-1, 1),
width,
height,
"bilinear",
"center",
).movedim(1, -1)
encode_pixels = pixels[:, :, :, :3]
if optional_cond_images is not None and 0 in optional_cond_indices:
# apply classical i2v conditioning on the first frame
idx_0 = optional_cond_indices.index(0)
encode_pixels = optional_cond_images[idx_0 : idx_0 + 1, :, :, :3]
t = vae.encode(encode_pixels)
latents["samples"][:, :, : t.shape[2]] = t
conditioning_latent_frames_mask = torch.ones(
(1, 1, latents["samples"].shape[2], 1, 1),
dtype=torch.float32,
device=latents["samples"].device,
)
conditioning_latent_frames_mask[:, :, : t.shape[2]] = 1.0 - strength
latents["noise_mask"] = conditioning_latent_frames_mask
if "noise_mask" not in latents:
conditioning_latent_frames_mask = torch.ones(
(1, 1, latents["samples"].shape[2], 1, 1),
dtype=torch.float32,
device=latents["samples"].device,
)
conditioning_latent_frames_mask[:, :, : t.shape[2]] = 1.0 - strength
latents["noise_mask"] = conditioning_latent_frames_mask
else:
latents["noise_mask"][:, :, : t.shape[2]] = 1.0 - strength
conditioning_latent_frames_mask = latents["noise_mask"]
else:
conditioning_latent_frames_mask = None
elif optional_cond_images is not None:
high_sigmas, rest_sigmas = SplitSigmas().get_sigmas(sigmas, guiding_start_step)
middle_sigmas, low_sigmas = SplitSigmas().get_sigmas(
rest_sigmas, guiding_end_step - guiding_start_step
)
if len(high_sigmas) > 1:
print("Denoising with no conditioning on sigmas: ", high_sigmas)
(_, new_latents) = SamplerCustomAdvanced().sample(
noise=noise,
guider=guider,
sampler=sampler,
sigmas=high_sigmas,
latent_image=latents,
)
if optional_cond_images is not None:
# add conditioning on keyframes with index > 0
for cond_image, cond_idx in zip(
optional_cond_images, optional_cond_indices
):
if cond_idx == 0:
# 0 is handled by classical i2v conditioning
continue
(
positive,
negative,
latents,
) = LTXVAddGuide().generate(
) = LTXVAddGuide.execute(
positive=positive,
negative=negative,
vae=vae,
@@ -228,25 +255,40 @@ class LTXVBaseSampler:
strength=optional_negative_index_strength,
)
guider = copy.copy(guider)
guider.set_conds(positive, negative)
# Denoise the latent video
print("Denoising with conditioning on sigmas: ", middle_sigmas)
(output_latents, denoised_output_latents) = SamplerCustomAdvanced().sample(
noise=noise,
guider=guider,
sampler=sampler,
sigmas=sigmas,
sigmas=middle_sigmas,
latent_image=latents,
)
# Clean up guides if image conditioning was used
positive, negative, denoised_output_latents = LTXVCropGuides().crop(
positive, negative, denoised_output_latents = LTXVCropGuides.execute(
positive=positive,
negative=negative,
latent=denoised_output_latents,
)
denoised_output_latents["noise_mask"] = conditioning_latent_frames_mask
if len(low_sigmas) > 1:
print(
"Denoising with no conditioning but with classical i2v noise mask on sigmas: ",
low_sigmas,
)
(_, denoised_output_latents) = SamplerCustomAdvanced().sample(
noise=noise,
guider=guider,
sampler=sampler,
sigmas=low_sigmas,
latent_image=denoised_output_latents,
)
return (denoised_output_latents, positive, negative)
@@ -254,7 +296,6 @@ class LTXVBaseSampler:
name="LTXVExtendSampler",
)
class LTXVExtendSampler:
@classmethod
def INPUT_TYPES(s):
return {
@@ -307,11 +348,25 @@ class LTXVExtendSampler:
"LATENT",
{"tooltip": "Optional latents to guide the sampling."},
),
"optional_cond_images": (
"IMAGE",
{"tooltip": "The images to use for conditioning the sampling."},
),
"optional_cond_indices": ("STRING",),
"cond_image_strength": (
"FLOAT",
{
"default": 1.0,
"min": 0.0,
"max": 1.0,
"tooltip": "The strength of the conditioning on the images.",
},
),
},
}
RETURN_TYPES = ("LATENT", "CONDITIONING", "CONDITIONING")
RETURN_NAMES = ("denoised_output", "positive", "negative")
RETURN_NAMES = ("denoised_video", "positive", "negative")
FUNCTION = "sample"
CATEGORY = "sampling"
@@ -328,22 +383,31 @@ class LTXVExtendSampler:
noise,
strength=0.5,
guiding_strength=1.0,
cond_image_strength=1.0,
optional_guiding_latents=None,
optional_cond_images=None,
optional_cond_indices=None,
optional_reference_latents=None,
optional_initialization_latents=None,
adain_factor=0.0,
optional_negative_index_latents=None,
optional_negative_index=-1,
optional_negative_index_strength=1.0,
guiding_start_step=0,
guiding_end_step=1000,
normalize_per_frame=False,
):
guider = copy.copy(guider)
guider.original_conds = copy.deepcopy(guider.original_conds)
try:
positive, negative = guider.raw_conds
except AttributeError:
raise ValueError(
"Guider does not have raw conds, cannot use it as a guider. "
"Please use STGGuiderAdvanced."
)
if optional_cond_indices is not None and optional_cond_images is not None:
optional_cond_indices = optional_cond_indices.split(",")
optional_cond_indices = [int(i) for i in optional_cond_indices]
assert len(optional_cond_indices) == len(
optional_cond_images
), "Number of optional cond images must match number of optional cond indices"
positive, negative = _get_raw_conds_from_guider(guider)
samples = latents["samples"]
batch, channels, frames, height, width = samples.shape
@@ -362,7 +426,7 @@ class LTXVExtendSampler:
)
if optional_initialization_latents is None:
new_latents = EmptyLTXVLatentVideo().generate(
new_latents = EmptyLTXVLatentVideo.execute(
width=width * width_scale_factor,
height=height * height_scale_factor,
length=overlap * time_scale_factor + num_new_frames,
@@ -389,6 +453,46 @@ class LTXVExtendSampler:
strength=strength,
)
high_sigmas, rest_sigmas = SplitSigmas().get_sigmas(sigmas, guiding_start_step)
middle_sigmas, low_sigmas = SplitSigmas().get_sigmas(
rest_sigmas, guiding_end_step - guiding_start_step
)
if optional_cond_images is not None:
print("Adding conditioning on keyframes")
for cond_image, cond_idx in zip(
optional_cond_images, optional_cond_indices
):
if optional_guiding_latents is not None and cond_idx % 8 == 1:
raise ValueError(
f"Conditioning image index {cond_idx} (relative to the in the temporal chunk) is a multiple of 8 + 1,"
"and guiding latents are used. Please provide other conditioning image indices"
)
(
positive,
negative,
new_latents,
) = LTXVAddGuide.execute(
positive=positive,
negative=negative,
vae=vae,
latent=new_latents,
image=cond_image.unsqueeze(0),
frame_idx=cond_idx,
strength=cond_image_strength,
)
if len(high_sigmas) > 1:
guider.set_conds(positive, negative)
print("Denoising with overlap conditioning only on sigmas: ", high_sigmas)
(_, new_latents) = SamplerCustomAdvanced().sample(
noise=noise,
guider=guider,
sampler=sampler,
sigmas=high_sigmas,
latent_image=new_latents,
)
if optional_guiding_latents is not None:
optional_guiding_latents = LTXVSelectLatents().select_latents(
optional_guiding_latents, overlap, -1
@@ -406,6 +510,7 @@ class LTXVExtendSampler:
latent_idx=last_overlap_latents["samples"].shape[2],
strength=guiding_strength,
)
if optional_negative_index_latents is not None:
(
positive,
@@ -421,37 +526,94 @@ class LTXVExtendSampler:
strength=optional_negative_index_strength,
)
guider = copy.copy(guider)
guider.set_conds(positive, negative)
# Denoise the latent video
print("Denoising with full conditioning on sigmas: ", middle_sigmas)
(output_latents, denoised_output_latents) = SamplerCustomAdvanced().sample(
noise=noise,
guider=guider,
sampler=sampler,
sigmas=sigmas,
sigmas=middle_sigmas,
latent_image=new_latents,
)
# Clean up guides if image conditioning was used
positive, negative, denoised_output_latents = LTXVCropGuides().crop(
positive, negative, denoised_output_latents = LTXVCropGuides.execute(
positive=positive,
negative=negative,
latent=denoised_output_latents,
)
if len(low_sigmas) > 1:
(
positive,
negative,
denoised_output_latents,
) = LTXVAddLatentGuide().generate(
vae=vae,
positive=positive,
negative=negative,
latent=denoised_output_latents,
guiding_latent=last_overlap_latents,
latent_idx=0,
strength=strength,
)
if optional_cond_images is not None:
print("Adding conditioning on keyframes")
for cond_image, cond_idx in zip(
optional_cond_images, optional_cond_indices
):
if optional_guiding_latents is not None and cond_idx % 8 == 1:
raise ValueError(
f"Conditioning image index {cond_idx} (relative to the in the temporal chunk) is a multiple of 8 + 1,"
"and guiding latents are used. Please provide other conditioning image indices"
)
(
positive,
negative,
denoised_output_latents,
) = LTXVAddGuide.execute(
positive=positive,
negative=negative,
vae=vae,
latent=denoised_output_latents,
image=cond_image.unsqueeze(0),
frame_idx=cond_idx,
strength=cond_image_strength,
)
guider.set_conds(positive, negative)
print(
"Denoising with overlap + keyframes conditioning only on sigmas: ",
low_sigmas,
)
(_, denoised_output_latents) = SamplerCustomAdvanced().sample(
noise=noise,
guider=guider,
sampler=sampler,
sigmas=low_sigmas,
latent_image=denoised_output_latents,
)
positive, negative, denoised_output_latents = LTXVCropGuides.execute(
positive=positive,
negative=negative,
latent=denoised_output_latents,
)
if optional_reference_latents is not None:
denoised_output_latents = LTXVAdainLatent().batch_normalize(
latents=denoised_output_latents,
reference=optional_reference_latents,
factor=adain_factor,
per_frame=normalize_per_frame,
)[0]
# drop first output latent as it's a reinterpreted 8-frame latent understood as a 1-frame latent
truncated_denoised_output_latents = LTXVSelectLatents().select_latents(
denoised_output_latents, 1, -1
)[0]
if optional_reference_latents is not None:
truncated_denoised_output_latents = LTXVAdainLatent().batch_normalize(
latents=truncated_denoised_output_latents,
reference=optional_reference_latents,
factor=adain_factor,
)[0]
# Fuse new frames with old ones by calling LinearOverlapLatentTransition
(latents,) = LinearOverlapLatentTransition().process(
latents, truncated_denoised_output_latents, overlap - 1, axis=2
@@ -484,7 +646,7 @@ class LTXVInContextSampler:
),
},
"optional": {
"optional_cond_image": (
"optional_cond_images": (
"IMAGE",
{
"tooltip": "The image to use for conditioning the sampling, if not provided, the sampling will be unconditioned (t2v setup). The image will be resized to the size of the first frame."
@@ -504,7 +666,7 @@ class LTXVInContextSampler:
}
RETURN_TYPES = ("LATENT", "CONDITIONING", "CONDITIONING")
RETURN_NAMES = ("denoised_output", "positive", "negative")
RETURN_NAMES = ("denoised_video", "positive", "negative")
FUNCTION = "sample"
CATEGORY = "sampling"
@@ -516,22 +678,31 @@ class LTXVInContextSampler:
sigmas,
noise,
guiding_latents,
optional_cond_image=None,
optional_cond_images=None,
optional_cond_indices=None,
num_frames=0,
optional_initialization_latents=None,
optional_negative_index_latents=None,
optional_negative_index=-1,
optional_negative_index_strength=1.0,
optional_cond_strength=1.0,
optional_guiding_strength=1.0,
cond_image_strength=1.0,
guiding_strength=1.0,
guiding_start_step=0,
guiding_end_step=1000,
):
try:
positive, negative = guider.raw_conds
except AttributeError:
raise ValueError(
"Guider does not have raw conds, cannot use it as a guider. "
"Please use STGGuiderAdvanced."
)
guider = copy.copy(guider)
guider.original_conds = copy.deepcopy(guider.original_conds)
if optional_cond_images is None:
optional_cond_indices = None
if optional_cond_indices is not None and optional_cond_images is not None:
optional_cond_indices = optional_cond_indices.split(",")
optional_cond_indices = [int(i) for i in optional_cond_indices]
assert len(optional_cond_indices) == len(
optional_cond_images
), "Number of optional cond images must match number of optional cond indices"
positive, negative = _get_raw_conds_from_guider(guider)
time_scale_factor, width_scale_factor, height_scale_factor = (
vae.downscale_index_formula
@@ -544,45 +715,40 @@ class LTXVInContextSampler:
if optional_initialization_latents is not None:
new_latents = optional_initialization_latents
else:
new_latents = EmptyLTXVLatentVideo().generate(
new_latents = EmptyLTXVLatentVideo.execute(
width=width * width_scale_factor,
height=height * height_scale_factor,
length=(frames - 1) * time_scale_factor + 1,
batch_size=1,
)[0]
if optional_cond_image is not None:
optional_cond_image = (
comfy.utils.common_upscale(
optional_cond_image.movedim(-1, 1),
width * width_scale_factor,
height * height_scale_factor,
"bilinear",
crop="disabled",
)
.movedim(1, -1)
.clamp(0, 1)
high_sigmas, rest_sigmas = SplitSigmas().get_sigmas(sigmas, guiding_start_step)
middle_sigmas, low_sigmas = SplitSigmas().get_sigmas(
rest_sigmas, guiding_end_step - guiding_start_step
)
if len(high_sigmas) > 1:
print(
"Denoising with keyframes only [if available] on sigmas: ",
high_sigmas,
)
(cond_image_latent,) = VAEEncode().encode(vae, optional_cond_image)
(
positive,
negative,
new_latents,
) = LTXVAddLatentGuide().generate(
vae=vae,
positive=positive,
negative=negative,
latent=new_latents,
guiding_latent=cond_image_latent,
latent_idx=0,
strength=optional_cond_strength,
(_, new_latents) = SamplerCustomAdvanced().sample(
noise=noise,
guider=guider,
sampler=sampler,
sigmas=high_sigmas,
latent_image=new_latents,
)
if optional_cond_image is not None:
if optional_cond_indices is not None and 0 in optional_cond_indices:
guiding_latents = LTXVSelectLatents().select_latents(
guiding_latents, 1, -1
)[0]
skip_one_guiding_latent = True
else:
skip_one_guiding_latent = False
print("Adding conditioning on guiding latents")
(
positive,
negative,
@@ -593,9 +759,31 @@ class LTXVInContextSampler:
negative=negative,
latent=new_latents,
guiding_latent=guiding_latents,
latent_idx=1 if optional_cond_image is not None else 0,
strength=optional_guiding_strength,
latent_idx=1 if skip_one_guiding_latent else 0,
strength=guiding_strength,
)
if optional_cond_images is not None:
print("Adding conditioning on keyframes", optional_cond_indices)
for cond_image, cond_idx in zip(
optional_cond_images, optional_cond_indices
):
if cond_idx % 8 == 1:
raise ValueError(
f"Conditioning image index {cond_idx} is a multiple of 8 + 1 and guiding latents are used. Please provide other cond image indices"
)
(
positive,
negative,
new_latents,
) = LTXVAddGuide.execute(
positive=positive,
negative=negative,
vae=vae,
latent=new_latents,
image=cond_image.unsqueeze(0),
frame_idx=cond_idx,
strength=cond_image_strength,
)
if optional_negative_index_latents is not None:
(
positive,
@@ -611,25 +799,44 @@ class LTXVInContextSampler:
strength=optional_negative_index_strength,
)
guider = copy.copy(guider)
guider.set_conds(positive, negative)
# Denoise the latent video
print("Denoising with full conditioning on sigmas: ", middle_sigmas)
(_, denoised_output_latents) = SamplerCustomAdvanced().sample(
noise=noise,
guider=guider,
sampler=sampler,
sigmas=sigmas,
sigmas=middle_sigmas,
latent_image=new_latents,
)
# Clean up guides if image conditioning was used
positive, negative, denoised_output_latents = LTXVCropGuides().crop(
positive, negative, denoised_output_latents = LTXVCropGuides.execute(
positive=positive,
negative=negative,
latent=denoised_output_latents,
)
if len(low_sigmas) > 1:
guider.set_conds(positive, negative)
print(
"Denoising with keyframes only [if available] conditioning on sigmas: ",
low_sigmas,
)
(_, denoised_output_latents) = SamplerCustomAdvanced().sample(
noise=noise,
guider=guider,
sampler=sampler,
sigmas=low_sigmas,
latent_image=denoised_output_latents,
)
positive, negative, denoised_output_latents = LTXVCropGuides.execute(
positive=positive,
negative=negative,
latent=denoised_output_latents,
)
return (denoised_output_latents, positive, negative)
@@ -690,7 +897,9 @@ class LinearOverlapLatentTransition:
]
combined_samples = torch.cat(parts, dim=axis)
combined_batch_index = torch.arange(0, combined_samples.shape[0])
combined_batch_index = torch.arange(0, combined_samples.shape[0]).to(
dtype=torch.float32
)
return (
{
embeddings_connector.py
+331
View File
@@ -0,0 +1,331 @@
import math
from typing import Optional
import comfy.ldm.common_dit
import torch
from comfy.ldm.lightricks.model import (
CrossAttention,
FeedForward,
generate_freq_grid_np,
interleaved_freqs_cis,
split_freqs_cis,
)
from torch import nn
from .pos_embedding_exp_values import POS_EMBEDDING_EXP_VALUES
class BasicTransformerBlock1D(nn.Module):
r"""
A basic Transformer block.
Parameters:
dim (`int`): The number of channels in the input and output.
num_attention_heads (`int`): The number of heads to use for multi-head attention.
attention_head_dim (`int`): The number of channels in each head.
dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use.
activation_fn (`str`, *optional*, defaults to `"geglu"`): Activation function to be used in feed-forward.
attention_bias (:
obj: `bool`, *optional*, defaults to `False`): Configure if the attentions should contain a bias parameter.
upcast_attention (`bool`, *optional*):
Whether to upcast the attention computation to float32. This is useful for mixed precision training.
norm_elementwise_affine (`bool`, *optional*, defaults to `True`):
Whether to use learnable elementwise affine parameters for normalization.
standardization_norm (`str`, *optional*, defaults to `"layer_norm"`): The type of pre-normalization to use. Can be `"layer_norm"` or `"rms_norm"`.
norm_eps (`float`, *optional*, defaults to 1e-5): Epsilon value for normalization layers.
qk_norm (`str`, *optional*, defaults to None):
Set to 'layer_norm' or `rms_norm` to perform query and key normalization.
final_dropout (`bool` *optional*, defaults to False):
Whether to apply a final dropout after the last feed-forward layer.
ff_inner_dim (`int`, *optional*): Dimension of the inner feed-forward layer. If not provided, defaults to `dim * 4`.
ff_bias (`bool`, *optional*, defaults to `True`): Whether to use bias in the feed-forward layer.
attention_out_bias (`bool`, *optional*, defaults to `True`): Whether to use bias in the attention output layer.
use_rope (`bool`, *optional*, defaults to `False`): Whether to use Rotary Position Embeddings (RoPE).
ffn_dim_mult (`int`, *optional*, defaults to 4): Multiplier for the inner dimension of the feed-forward layer.
"""
def __init__(
self,
dim,
n_heads,
d_head,
context_dim=None,
attn_precision=None,
dtype=None,
device=None,
operations=None,
):
super().__init__()
# Define 3 blocks. Each block has its own normalization layer.
# 1. Self-Attn
self.attn1 = CrossAttention(
query_dim=dim,
heads=n_heads,
dim_head=d_head,
context_dim=None,
dtype=dtype,
device=device,
operations=operations,
)
# 3. Feed-forward
self.ff = FeedForward(
dim,
dim_out=dim,
glu=True,
dtype=dtype,
device=device,
operations=operations,
)
def forward(self, hidden_states, attention_mask=None, pe=None) -> torch.FloatTensor:
# Notice that normalization is always applied before the real computation in the following blocks.
# 1. Normalization Before Self-Attention
norm_hidden_states = comfy.ldm.common_dit.rms_norm(hidden_states)
norm_hidden_states = norm_hidden_states.squeeze(1)
# 2. Self-Attention
attn_output = self.attn1(norm_hidden_states, mask=attention_mask, pe=pe)
hidden_states = attn_output + hidden_states
if hidden_states.ndim == 4:
hidden_states = hidden_states.squeeze(1)
# 3. Normalization before Feed-Forward
norm_hidden_states = comfy.ldm.common_dit.rms_norm(hidden_states)
# 4. Feed-forward
ff_output = self.ff(norm_hidden_states)
hidden_states = ff_output + hidden_states
if hidden_states.ndim == 4:
hidden_states = hidden_states.squeeze(1)
return hidden_states
class Embeddings1DConnector(nn.Module):
_supports_gradient_checkpointing = True
def __init__(
self,
in_channels=128,
cross_attention_dim=2048,
attention_head_dim=128,
num_attention_heads=30,
num_layers=2,
positional_embedding_theta=10000.0,
positional_embedding_max_pos=[1],
causal_temporal_positioning=False,
num_learnable_registers: Optional[int] = 128,
dtype=None,
device=None,
operations=None,
split_rope=False,
double_precision_rope=False,
**kwargs,
):
super().__init__()
self.dtype = dtype
self.out_channels = in_channels
self.num_attention_heads = num_attention_heads
self.inner_dim = num_attention_heads * attention_head_dim
self.causal_temporal_positioning = causal_temporal_positioning
self.positional_embedding_theta = positional_embedding_theta
self.positional_embedding_max_pos = positional_embedding_max_pos
self.split_rope = split_rope
self.double_precision_rope = double_precision_rope
self.transformer_1d_blocks = nn.ModuleList(
[
BasicTransformerBlock1D(
self.inner_dim,
num_attention_heads,
attention_head_dim,
context_dim=cross_attention_dim,
dtype=dtype,
device=device,
operations=operations,
)
for _ in range(num_layers)
]
)
inner_dim = num_attention_heads * attention_head_dim
self.num_learnable_registers = num_learnable_registers
if self.num_learnable_registers:
self.learnable_registers = nn.Parameter(
torch.rand(
self.num_learnable_registers, inner_dim, dtype=dtype, device=device
)
* 2.0
- 1.0
)
def get_fractional_positions(self, indices_grid):
fractional_positions = torch.stack(
[
indices_grid[:, i] / self.positional_embedding_max_pos[i]
for i in range(1)
],
dim=-1,
)
return fractional_positions
def precompute_freqs(self, indices_grid, spacing):
source_dtype = indices_grid.dtype
dtype = (
torch.float32
if source_dtype in (torch.bfloat16, torch.float16)
else source_dtype
)
fractional_positions = self.get_fractional_positions(indices_grid)
indices = (
generate_freq_grid_np(
self.positional_embedding_theta,
indices_grid.shape[1],
self.inner_dim,
)
if self.double_precision_rope
else self.generate_freq_grid(spacing, dtype, fractional_positions.device)
).to(device=fractional_positions.device)
if spacing == "exp_2":
freqs = (
(indices * fractional_positions.unsqueeze(-1))
.transpose(-1, -2)
.flatten(2)
)
else:
freqs = (
(indices * (fractional_positions.unsqueeze(-1) * 2 - 1))
.transpose(-1, -2)
.flatten(2)
)
return freqs
def generate_freq_grid(self, spacing, dtype, device):
dim = self.inner_dim
theta = self.positional_embedding_theta
n_pos_dims = 1
n_elem = 2 * n_pos_dims # 2 for cos and sin e.g. x 3 = 6
start = 1
end = theta
if spacing == "exp":
# Workaround float precision issues when calculating positional embeddings exponents to ensure compatibility with JAX.
indices = torch.tensor(POS_EMBEDDING_EXP_VALUES, dtype=dtype, device=device)
elif spacing == "exp_2":
indices = 1.0 / theta ** (torch.arange(0, dim, n_elem, device=device) / dim)
indices = indices.to(dtype=dtype)
elif spacing == "linear":
indices = torch.linspace(
start, end, dim // n_elem, device=device, dtype=dtype
)
elif spacing == "sqrt":
indices = torch.linspace(
start**2, end**2, dim // n_elem, device=device, dtype=dtype
).sqrt()
indices = indices * math.pi / 2
return indices
def precompute_freqs_cis(self, indices_grid, spacing="exp"):
dim = self.inner_dim
n_elem = 2 # 2 because of cos and sin
freqs = self.precompute_freqs(indices_grid, spacing)
if self.split_rope:
expected_freqs = dim // 2
current_freqs = freqs.shape[-1]
pad_size = expected_freqs - current_freqs
cos_freq, sin_freq = split_freqs_cis(
freqs, pad_size, self.num_attention_heads
)
else:
cos_freq, sin_freq = interleaved_freqs_cis(freqs, dim % n_elem)
return cos_freq.to(self.dtype), sin_freq.to(self.dtype), self.split_rope
def forward(
self,
hidden_states: torch.Tensor,
attention_mask: Optional[torch.Tensor] = None,
):
"""
The [`Transformer2DModel`] forward method.
Args:
hidden_states (`torch.LongTensor` of shape `(batch size, num latent pixels)` if discrete, `torch.FloatTensor` of shape `(batch size, channel, height, width)` if continuous):
Input `hidden_states`.
indices_grid (`torch.LongTensor` of shape `(batch size, 3, num latent pixels)`):
attention_mask ( `torch.Tensor`, *optional*):
An attention mask of shape `(batch, key_tokens)` is applied to `encoder_hidden_states`. If `1` the mask
is kept, otherwise if `0` it is discarded. Mask will be converted into a bias, which adds large
negative values to the attention scores corresponding to "discard" tokens.
Returns:
If `return_dict` is True, an [`~models.transformer_2d.Transformer2DModelOutput`] is returned, otherwise a
`tuple` where the first element is the sample tensor.
"""
# 1. Input
if self.num_learnable_registers:
# replace all padded tokens with learnable registers
assert (
hidden_states.shape[1] % self.num_learnable_registers == 0
), f"Hidden states sequence length {hidden_states.shape[1]} must be divisible by num_learnable_registers {self.num_learnable_registers}."
num_registers_duplications = (
hidden_states.shape[1] // self.num_learnable_registers
)
learnable_registers = torch.tile(
self.learnable_registers, (num_registers_duplications, 1)
)
attention_mask_binary = (
attention_mask.squeeze(1).squeeze(1).unsqueeze(-1) >= -9000.0
).int()
non_zero_hidden_states = hidden_states[
:, attention_mask_binary.squeeze().bool(), :
]
non_zero_nums = non_zero_hidden_states.shape[1]
pad_length = hidden_states.shape[1] - non_zero_nums
adjusted_hidden_states = torch.nn.functional.pad(
non_zero_hidden_states, pad=(0, 0, 0, pad_length), value=0
)
flipped_mask = torch.flip(attention_mask_binary, dims=[1])
hidden_states = (
flipped_mask * adjusted_hidden_states
+ (1 - flipped_mask) * learnable_registers
)
attention_mask = torch.full_like(
attention_mask,
0.0,
dtype=attention_mask.dtype,
device=attention_mask.device,
)
indices_grid = torch.arange(
hidden_states.shape[1], dtype=torch.float32, device=hidden_states.device
)
indices_grid = indices_grid[None, None, :]
freqs_cis = self.precompute_freqs_cis(indices_grid)
# 2. Blocks
for block_idx, block in enumerate(self.transformer_1d_blocks):
hidden_states = block(
hidden_states, attention_mask=attention_mask, pe=freqs_cis
)
# 3. Output
# if self.output_scale is not None:
# hidden_states = hidden_states / self.output_scale
hidden_states = comfy.ldm.common_dit.rms_norm(hidden_states)
return hidden_states, attention_mask
example_workflows/13b-distilled/ltxv-13b-dist-i2v-base-fp8.json
-3281
View File
File diff suppressed because it is too large Load Diff
example_workflows/13b-distilled/ltxv-13b-dist-i2v-base.json
-3211
View File
File diff suppressed because it is too large Load Diff
example_workflows/13b-distilled/ltxv-13b-dist-i2v-extend.json
-3739
View File
File diff suppressed because it is too large Load Diff
example_workflows/13b-distilled/ltxv-13b-dist-i2v-keyframes.json
-3466
View File
File diff suppressed because it is too large Load Diff
example_workflows/LTX-2_I2V_Distilled_wLora.json
+3282
View File
File diff suppressed because it is too large Load Diff
example_workflows/LTX-2_I2V_Full_wLora.json
+3415
View File
File diff suppressed because it is too large Load Diff
example_workflows/LTX-2_ICLoRA_All_Distilled.json
+13892
View File
File diff suppressed because it is too large Load Diff
example_workflows/LTX-2_T2V_Distilled_wLora.json
+2835
View File
File diff suppressed because it is too large Load Diff
example_workflows/LTX-2_T2V_Full_wLora.json
+2977
View File
File diff suppressed because it is too large Load Diff
example_workflows/low_level/ltxvideo-i2v-distilled.json → example_workflows/LTX-2_V2V_Detailer.json
+875 -777
View File
File diff suppressed because it is too large Load Diff
example_workflows/assets/base model image.png
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6557896dab6ca788224de10cf66bb3b46fce990b921f4bb9783c2f43de77c209
size 1761966
example_workflows/assets/buildings ff.png
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0f32314bb85cde1b2fcc978a71d0fd93242c1132c397989a3fa1f4486574a3ff
size 1760295
example_workflows/assets/buildings.mp4
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4cfb677a93331ab9614c820ab4d7436eac0f54a4685a6bf8f9083000ecc66f6f
size 6625560
example_workflows/assets/distilled image.png
+3
View File
@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:64a49c9e6d5c8d9426afa1b3b3fdbac6cbf491e7e65cc21bdf2c677743a3982d
size 1130439
example_workflows/chimpanzee.webp
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 80 KiB

example_workflows/depth-surfing.mp4
-3
View File
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d177d33a43e5f7b87dfa24726f55a0aa698f06d7dad255b759560d1b23e2e3be
size 1481235
example_workflows/ic_lora/ic-lora.json
-6417
View File
File diff suppressed because it is too large Load Diff
example_workflows/ic_lora/man_walking.mp4
-3
View File
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:346eeff236979002921588066d80fff8bd2e0cf51d34b1061aac8d7bac3554b7
size 3604332
example_workflows/ic_lora/robot.png
-3
View File
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:fd423a6c09f33e2c1bcbc147f64502fa677b976bd7d84c3c51b87d2c4122eb03
size 1347335
example_workflows/low_level/end.jpg
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 474 KiB

example_workflows/low_level/fox.jpg
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 431 KiB

example_workflows/low_level/jeep.mp4
-3
View File
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5b8ac2148bdf1092f4342eb4f08961458a9e22743e8896f5dbcf749166f4f526
size 1582922
example_workflows/low_level/ltxvideo-first-sequence-conditioning.json
-2553
View File
File diff suppressed because it is too large Load Diff
example_workflows/low_level/ltxvideo-frame-interpolation.json
-1892
View File
File diff suppressed because it is too large Load Diff
example_workflows/low_level/ltxvideo-i2v.json
-1768
View File
File diff suppressed because it is too large Load Diff
example_workflows/low_level/ltxvideo-last-sequence-conditioning.json
-2422
View File
File diff suppressed because it is too large Load Diff
example_workflows/low_level/ltxvideo-t2v.json
-1137
View File
File diff suppressed because it is too large Load Diff
example_workflows/low_level/moto.png
-3
View File
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8681ff86edb43d0a12472175d5503a590910b24d5ec5d5d5b78eed079960eac0
size 998115
example_workflows/low_level/shrek2.jpg
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 349 KiB

example_workflows/low_level/start.jpg
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 570 KiB

example_workflows/ltxv-13b-i2v-base-fp8.json
-2504
View File
File diff suppressed because it is too large Load Diff
example_workflows/ltxv-13b-i2v-base.json
-2444
View File
File diff suppressed because it is too large Load Diff
example_workflows/ltxv-13b-i2v-extend.json
-2875
View File
File diff suppressed because it is too large Load Diff
example_workflows/ltxv-13b-i2v-keyframes.json
-2716
View File
File diff suppressed because it is too large Load Diff
example_workflows/ltxv-13b-i2v-long-multi-prompt.json
-4773
View File
File diff suppressed because it is too large Load Diff
example_workflows/ltxv-13b-i2v-mixed-multiscale.json
-3508
View File
File diff suppressed because it is too large Load Diff
example_workflows/ltxv-13b-upscale.json
-2604
View File
File diff suppressed because it is too large Load Diff
example_workflows/ltxv-13b-v2v-long-depth.json
-4864
View File
File diff suppressed because it is too large Load Diff
example_workflows/tricks/ltxvideo-flow-edit.json
-2962
View File
File diff suppressed because it is too large Load Diff
example_workflows/tricks/ltxvideo-flow-edit.png
-3
View File
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:041f3820e0ab4109137d743b3f6b8e2dea0cddcad194aac08d3baec7571200f9
size 1841100
example_workflows/tricks/ltxvideo-rf-edit.json
-3877
View File
File diff suppressed because it is too large Load Diff
example_workflows/tricks/ltxvideo-rf-edit.png
-3
View File
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9d2b002f12155d0b929366ad38561680a5bbaecea1925e5655658b46d281baf9
size 2441562
example_workflows/tricks/ref.png
-3
View File
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3726ed083a5c52873d29adab87e8fe0a65c7e390beaf8dfcf23e5c9091d6da73
size 521574
example_workflows/tricks/shot.mp4
-3
View File
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:01d44bb728e04879a5513a2c33bdd47d443bf901b33d3c1445918e75ba8b78b1
size 678605
example_workflows/tricks/shot2.mp4
-3
View File
@@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5432150d5477d19586b8d8049ca214038f886d3def9117830cf2af58879c48b5
size 31584408
example_workflows/tricks/shrek2.jpg
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 349 KiB

example_workflows/tricks/shrek3.jpg
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 342 KiB

film_grain.py
-60
View File
@@ -1,60 +0,0 @@
from typing import Tuple
import comfy
import torch
from .nodes_registry import comfy_node
@comfy_node(
name="LTXVFilmGrain",
)
class LTXVFilmGrain:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"images": ("IMAGE",),
"grain_intensity": (
"FLOAT",
{"default": 0.1, "min": 0.0, "max": 1.0, "step": 0.01},
),
"saturation": (
"FLOAT",
{"default": 0.5, "min": 0.0, "max": 1.0, "step": 0.01},
),
},
"optional": {},
}
RETURN_TYPES = ("IMAGE",)
FUNCTION = "add_film_grain"
CATEGORY = "effects"
DESCRIPTION = "Adds film grain to the image."
def add_film_grain(
self, images: torch.Tensor, grain_intensity: float, saturation: float
) -> Tuple[torch.Tensor]:
if grain_intensity < 0 or grain_intensity > 1:
raise ValueError("Grain intensity must be between 0 and 1.")
device = comfy.model_management.get_torch_device()
images = images.to(device)
grain = torch.zeros(images[0:1].shape, device=device)
# Process images in-place to reduce memory usage
for i in range(images.shape[0]):
# Generate grain for single image
torch.randn(grain.shape, device=device, out=grain)
grain[:, :, :, 0] *= 2
grain[:, :, :, 2] *= 3
grain = grain * saturation + grain[:, :, :, 1].unsqueeze(3).expand(
-1, -1, -1, 3
) * (1 - saturation)
# Blend the grain with the image in-place
images[i : i + 1].add_(grain_intensity * grain)
images[i : i + 1].clamp_(0, 1)
images = images.to(comfy.model_management.intermediate_device())
return (images,)
gemma_encoder.py
+714
View File
@@ -0,0 +1,714 @@
import json
import logging
from pathlib import Path
from typing import Optional
import comfy.model_management
import comfy.ops
import comfy.sd
import comfy.supported_models_base
import folder_paths
import safetensors
import torch
from comfy.ldm.lightricks.model import LTXFrequenciesPrecision, LTXRopeType
from comfy.utils import load_torch_file
from einops import rearrange
from PIL import Image
from transformers import (
AutoImageProcessor,
AutoTokenizer,
Gemma3ForConditionalGeneration,
Gemma3Processor,
)
from .embeddings_connector import Embeddings1DConnector
from .nodes_registry import comfy_node
logger = logging.getLogger(__name__)
PREFIX_BASE = "model.diffusion_model."
def _load_system_prompt(filename: str) -> str:
"""Load system prompt from file at module level."""
try:
prompt_path = Path(__file__).parent / "system_prompts" / filename
if prompt_path.exists():
return prompt_path.read_text(encoding="utf-8").strip()
except Exception as e:
logger.warning(f"Could not load {filename}: {e}")
return ""
DEFAULT_T2V_SYSTEM_PROMPT = _load_system_prompt("gemma_t2v_system_prompt.txt")
DEFAULT_I2V_SYSTEM_PROMPT = _load_system_prompt("gemma_i2v_system_prompt.txt")
def tensor_to_pil(tensor: torch.Tensor) -> Image.Image:
"""Convert ComfyUI image tensor to PIL Image."""
if tensor.dim() == 4:
tensor = tensor[0]
numpy_image = (tensor.cpu().numpy() * 255).astype("uint8")
return Image.fromarray(numpy_image)
def load_video_embeddings_connector(ltxv_path, dtype=torch.bfloat16):
sd, metadata = load_torch_file(str(ltxv_path), return_metadata=True)
config = json.loads(metadata.get("config", "{}"))
transformer_config = config.get("transformer", {})
rope_type = LTXRopeType.from_dict(transformer_config)
frequencies_precision = LTXFrequenciesPrecision.from_dict(transformer_config)
pe_max_pos = transformer_config.get("connector_positional_embedding_max_pos", [1])
sd_keys = list(sd.keys())
video_only_connector_prefix = f"{PREFIX_BASE}embeddings_connector."
av_connector_prefix = f"{PREFIX_BASE}video_embeddings_connector."
prefix = (
av_connector_prefix
if f"{PREFIX_BASE}audio_adaln_single.linear.weight" in sd_keys
else video_only_connector_prefix
)
return load_embeddings_connector(
sd, prefix, dtype, rope_type, frequencies_precision, pe_max_pos
)
def load_audio_embeddings_connector(ltxv_path, dtype=torch.bfloat16):
sd, metadata = load_torch_file(str(ltxv_path), return_metadata=True)
config = json.loads(metadata.get("config", "{}"))
transformer_config = config.get("transformer", {})
rope_type = LTXRopeType.from_dict(transformer_config)
frequencies_precision = LTXFrequenciesPrecision.from_dict(transformer_config)
pe_max_pos = transformer_config.get("connector_positional_embedding_max_pos", [1])
return load_embeddings_connector(
sd,
f"{PREFIX_BASE}audio_embeddings_connector.",
dtype,
rope_type,
frequencies_precision,
pe_max_pos,
)
def load_embeddings_connector(
sd,
connector_prefix,
dtype=torch.bfloat16,
rope_type=LTXRopeType.INTERLEAVED,
frequencies_precision=LTXFrequenciesPrecision.FLOAT32,
pe_max_pos=None,
):
sd_connector = {
k[len(connector_prefix) :]: v
for k, v in sd.items()
if k.startswith(connector_prefix)
}
if len(sd_connector) == 0:
return None
operations = comfy.ops.pick_operations(dtype, dtype, disable_fast_fp8=True)
connector = Embeddings1DConnector(
dtype=dtype,
operations=operations,
positional_embedding_max_pos=pe_max_pos if pe_max_pos is not None else [1],
split_rope=rope_type == LTXRopeType.SPLIT,
double_precision_rope=frequencies_precision == LTXFrequenciesPrecision.FLOAT64,
)
connector.load_state_dict(sd_connector)
return connector
def load_proj_matrix_from_ltxv(ltxv_path: Path, prefix=""):
with safetensors.safe_open(ltxv_path, framework="pt", device="cpu") as f:
keys = filter(lambda key: key.startswith(prefix), f.keys())
sd = {k.removeprefix(prefix): f.get_tensor(k) for k in keys if k in f.keys()}
if not sd:
return None
model = GemmaFeaturesExtractorProjLinear()
model.load_state_dict(sd)
return model
def load_proj_matrix_from_checkpoint(checkpoint_path: Path):
"""
Load model weights from a checkpoint file.
:param checkpoint_path: Path to the checkpoint file.
"""
model = GemmaFeaturesExtractorProjLinear()
loaded_state_dict = load_torch_file(str(checkpoint_path), return_metadata=False)
if "aggregate_embed.weight" not in loaded_state_dict:
raise ValueError(
f"Checkpoint {checkpoint_path} does not contain 'aggregate_embed.weight'."
)
model.load_state_dict(loaded_state_dict)
return model
class LTXVGemmaTokenizer:
def __init__(self, tokenizer_path: str, max_length: int = 1024):
self.tokenizer = AutoTokenizer.from_pretrained(
tokenizer_path, local_files_only=True, model_max_length=max_length
)
# Gemma expects left padding for chat-style prompts; for plain text it doesn't matter much.
self.tokenizer.padding_side = "left"
if self.tokenizer.pad_token is None:
self.tokenizer.pad_token = self.tokenizer.eos_token
self.max_length = max_length
def tokenize_with_weights(self, text: str, return_word_ids: bool = False):
text = text.strip()
encoded = self.tokenizer(
text,
padding="max_length",
max_length=self.max_length,
truncation=True,
return_tensors="pt",
)
input_ids = encoded.input_ids
attention_mask = encoded.attention_mask
tuples = [
(token_id, attn, i)
for i, (token_id, attn) in enumerate(zip(input_ids[0], attention_mask[0]))
]
out = {"gemma": tuples}
if not return_word_ids:
out = {k: [(t, w) for t, w, _ in v] for k, v in out.items()}
return out
class GemmaFeaturesExtractorProjLinear(torch.nn.Module):
def __init__(self):
super().__init__()
self.aggregate_embed = torch.nn.Linear(3840 * 49, 3840, bias=False)
def forward(self, x):
return self.aggregate_embed(x)
class LTXVGemmaTextEncoderModel(torch.nn.Module):
def __init__(
self,
model: Gemma3ForConditionalGeneration,
feature_extractor_linear: GemmaFeaturesExtractorProjLinear,
embeddings_connector: Embeddings1DConnector,
audio_embeddings_connector: Embeddings1DConnector | None = None,
processor: Gemma3Processor | None = None,
dtype=torch.bfloat16,
device="cpu",
):
super().__init__()
self.model = model
self.processor = processor
self.feature_extractor_linear = feature_extractor_linear.to(dtype=dtype)
self.embeddings_connector = embeddings_connector.to(dtype=dtype)
self.audio_embeddings_connector = (
audio_embeddings_connector.to(dtype=dtype)
if audio_embeddings_connector is not None
else None
)
self.dtypes = set([dtype])
# Cache an estimate of memory required to load/keep the model on device
# weights size + small overhead
self._model_memory_required = (
comfy.model_management.module_size(self.model) + 256 * 1024 * 1024
)
def set_clip_options(self, options):
pass
def reset_clip_options(self):
pass
@staticmethod
def norm_and_concat_padded_batch(
encoded_text: torch.Tensor,
sequence_lengths: torch.Tensor,
padding_side: str = "right",
) -> torch.Tensor:
"""
Normalize a 4D tensor [B, T, D, L] per sample and per layer, using sequence_lengths to mask.
Returns [B, T, D * L] tensor with original padding preserved.
Args:
encoded_text: 4D tensor [B, T, D, L]
sequence_lengths: 1D tensor [B] with actual sequence lengths
padding_side: "left" or "right" to indicate which side has padding
"""
B, T, D, L = encoded_text.shape
device = encoded_text.device
# Build mask: [B, T, 1, 1]
token_indices = torch.arange(T, device=device)[None, :] # [1, T]
if padding_side == "right":
# For right padding, valid tokens are from 0 to sequence_length-1
mask = token_indices < sequence_lengths[:, None] # [B, T]
elif padding_side == "left":
# For left padding, valid tokens are from (T - sequence_length) to T-1
start_indices = T - sequence_lengths[:, None] # [B, 1]
mask = token_indices >= start_indices # [B, T]
else:
raise ValueError(
f"padding_side must be 'left' or 'right', got {padding_side}"
)
mask = rearrange(mask, "b t -> b t 1 1")
# Compute masked mean: [B, 1, 1, L]
masked = encoded_text.masked_fill(~mask, 0.0)
# denom = mask.sum(dim=(1, 2), keepdim=True) # [B, 1, 1, 1]
denom = (sequence_lengths * D).view(B, 1, 1, 1)
mean = masked.sum(dim=(1, 2), keepdim=True) / (denom + 1e-6)
# Compute masked min/max: [B, 1, 1, L]
x_min = encoded_text.masked_fill(~mask, float("inf")).amin(
dim=(1, 2), keepdim=True
)
x_max = encoded_text.masked_fill(~mask, float("-inf")).amax(
dim=(1, 2), keepdim=True
)
range_ = x_max - x_min
# Normalize only the valid tokens
normed = 8 * (encoded_text - mean) / (range_ + 1e-6)
# concat to be [Batch, T, D * L] - this preserves the original structure
normed = normed.reshape(B, T, -1) # [B, T, D * L]
# Apply mask to preserve original padding (set padded positions to 0)
mask_flattened = rearrange(mask, "b t 1 1 -> b t 1").expand(-1, -1, D * L)
normed = normed.masked_fill(~mask_flattened, 0.0)
return normed
def forward(self, input_ids, attention_mask, padding_side="right"):
outputs = self.model(
input_ids=input_ids,
attention_mask=attention_mask,
output_hidden_states=True,
)
encoded_text_features = torch.stack(outputs.hidden_states, dim=-1)
encoded_text_features_dtype = encoded_text_features.dtype
sequence_lengths = attention_mask.sum(dim=-1)
normed_concated_encoded_text_features = (
LTXVGemmaTextEncoderModel.norm_and_concat_padded_batch(
encoded_text_features, sequence_lengths, padding_side=padding_side
)
)
projected = self.feature_extractor_linear(
normed_concated_encoded_text_features.to(encoded_text_features_dtype)
)
return projected
def encode_token_weights(self, token_weight_pairs):
token_pairs = token_weight_pairs["gemma"]
input_ids = torch.tensor(
[[t[0] for t in token_pairs]], device=self.model.device
)
attention_mask = torch.tensor(
[[w[1] for w in token_pairs]], device=self.model.device
)
self.to(self.model.device)
encoded_input = self(input_ids, attention_mask, padding_side="left")
# convert attention mask to format embeddings connector expects.
connector_attention_mask = (attention_mask - 1).to(encoded_input.dtype).reshape(
(attention_mask.shape[0], 1, -1, attention_mask.shape[-1])
) * torch.finfo(encoded_input.dtype).max
encoded, encoded_connector_attention_mask = self.embeddings_connector(
encoded_input,
connector_attention_mask,
)
# restore the mask values to int64
attention_mask = (encoded_connector_attention_mask < 0.000001).to(torch.int64)
attention_mask = attention_mask.reshape([encoded.shape[0], encoded.shape[1], 1])
encoded = encoded * attention_mask
if self.audio_embeddings_connector is not None:
encoded_for_audio, _ = self.audio_embeddings_connector(
encoded_input, connector_attention_mask
)
encoded = torch.concat(
[encoded, encoded_for_audio], dim=len(encoded.shape) - 1
)
return encoded, None, {"attention_mask": attention_mask.squeeze(-1)}
def load_sd(self, sd):
return self.model.load_state_dict(sd, strict=False)
def memory_required(self, input_shape):
# Return a conservative estimate in bytesed(input_shape)
return self._model_memory_required
@staticmethod
def _cat_with_padding(
tensor: torch.Tensor,
padding_length: int,
value: int | float,
) -> torch.Tensor:
"""Concatenate a tensor with a padding tensor of the given value."""
return torch.cat(
[
tensor,
torch.full(
(1, padding_length),
value,
dtype=tensor.dtype,
device=tensor.device,
),
],
dim=1,
)
def _pad_inputs_for_attention_alignment(self, model_inputs, alignment: int = 8):
"""Pad sequence length to multiple of alignment for Flash Attention compatibility.
Flash Attention within SDPA requires sequence lengths aligned to 8 bytes.
This pads input_ids, attention_mask, and token_type_ids (if present) to prevent
'p.attn_bias_ptr is not correctly aligned' errors.
"""
seq_len = model_inputs.input_ids.shape[1]
padded_len = ((seq_len + alignment - 1) // alignment) * alignment
padding_length = padded_len - seq_len
if padding_length > 0:
pad_token_id = (
self.processor.tokenizer.pad_token_id
if self.processor.tokenizer.pad_token_id is not None
else 0
)
model_inputs["input_ids"] = self._cat_with_padding(
model_inputs.input_ids, padding_length, pad_token_id
)
model_inputs["attention_mask"] = self._cat_with_padding(
model_inputs.attention_mask, padding_length, 0
)
if (
"token_type_ids" in model_inputs
and model_inputs["token_type_ids"] is not None
):
model_inputs["token_type_ids"] = self._cat_with_padding(
model_inputs["token_type_ids"], padding_length, 0
)
return model_inputs
def _enhance(
self,
messages: list,
image: Optional[Image.Image] = None,
max_new_tokens: int = 512,
seed: int = 42,
) -> str:
"""Common enhancement logic for both T2V and I2V modes."""
if self.processor is None:
raise ValueError("Processor not loaded - enhancement not available")
text = self.processor.tokenizer.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
model_inputs = self.processor(
text=text,
images=image,
return_tensors="pt",
).to(self.model.device)
model_inputs = self._pad_inputs_for_attention_alignment(model_inputs)
with torch.inference_mode(), torch.random.fork_rng(devices=[self.model.device]):
torch.manual_seed(seed)
outputs = self.model.generate(
**model_inputs,
max_new_tokens=max_new_tokens,
do_sample=True,
temperature=0.7,
)
generated_ids = outputs[0][len(model_inputs.input_ids[0]) :]
enhanced_prompt = self.processor.tokenizer.decode(
generated_ids, skip_special_tokens=True
)
return enhanced_prompt
def enhance_t2v(
self,
prompt: str,
system_prompt: str,
max_new_tokens: int = 512,
seed: int = 42,
) -> str:
"""Enhance a text prompt for T2V generation."""
messages = [
{"role": "system", "content": system_prompt},
{"role": "user", "content": f"User Raw Input Prompt: {prompt}."},
]
return self._enhance(messages, max_new_tokens=max_new_tokens, seed=seed)
def enhance_i2v(
self,
prompt: str,
image: Image.Image,
system_prompt: str,
max_new_tokens: int = 512,
seed: int = 42,
) -> str:
"""Enhance a text prompt for I2V generation using a reference image."""
messages = [
{"role": "system", "content": system_prompt},
{
"role": "user",
"content": [
{"type": "image"},
{"type": "text", "text": f"User Raw Input Prompt: {prompt}."},
],
},
]
return self._enhance(
messages, image=image, max_new_tokens=max_new_tokens, seed=seed
)
def ltxv_gemma_tokenizer(tokenizer_path, max_length=256):
class _LTXVGemmaTokenizer(LTXVGemmaTokenizer):
def __init__(self, embedding_directory=None, tokenizer_data={}):
super().__init__(tokenizer_path, max_length=max_length)
return _LTXVGemmaTokenizer
def ltxv_gemma_clip(encoder_path, ltxv_path, processor=None, dtype=None):
class _LTXVGemmaTextEncoderModel(LTXVGemmaTextEncoderModel):
def __init__(self, device="cpu", dtype=dtype, model_options={}):
dtype = torch.bfloat16 # TODO: make this configurable
gemma_model = Gemma3ForConditionalGeneration.from_pretrained(
encoder_path,
local_files_only=True,
torch_dtype=dtype,
)
feature_extractor_linear = load_proj_matrix_from_ltxv(
ltxv_path,
"text_embedding_projection.",
)
if feature_extractor_linear is None:
feature_extractor_linear = load_proj_matrix_from_checkpoint(
encoder_path / "proj_linear.safetensors"
)
embeddings_connector = load_video_embeddings_connector(ltxv_path)
audio_embeddings_connector = load_audio_embeddings_connector(ltxv_path)
super().__init__(
model=gemma_model,
feature_extractor_linear=feature_extractor_linear,
embeddings_connector=embeddings_connector,
audio_embeddings_connector=audio_embeddings_connector,
processor=processor,
dtype=dtype,
device=device,
)
return _LTXVGemmaTextEncoderModel
def find_matching_dir(root_path: str, pattern: str) -> str:
"""
Recursively search for files matching a glob pattern and return the parent directory of the first match.
"""
matches = list(Path(root_path).rglob(pattern))
if not matches:
raise FileNotFoundError(
f"No files matching pattern '{pattern}' found under {root_path}"
)
return str(matches[0].parent)
@comfy_node(name="LTXVGemmaCLIPModelLoader", description="Gemma 3 Model Loader")
class LTXVGemmaCLIPModelLoader:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"gemma_path": (
folder_paths.get_filename_list("text_encoders"),
{"tooltip": "The name of the text encoder model to load."},
),
"ltxv_path": (
folder_paths.get_filename_list("checkpoints"),
{"tooltip": "The name of the ltxv model to load."},
),
"max_length": (
"INT",
{"default": 1024, "min": 16, "max": 131072, "step": 8},
),
}
}
RETURN_TYPES = ("CLIP",)
RETURN_NAMES = ("clip",)
FUNCTION = "load_model"
CATEGORY = "lightricks/LTXV"
TITLE = "LTXV Gemma CLIP Loader"
OUTPUT_NODE = False
def load_model(self, gemma_path: str, ltxv_path: str, max_length: int):
path = Path(folder_paths.get_full_path("text_encoders", gemma_path))
model_root = path.parents[1]
tokenizer_path = Path(find_matching_dir(model_root, "tokenizer.model"))
gemma_model_path = Path(find_matching_dir(model_root, "model*.safetensors"))
processor_path = Path(find_matching_dir(model_root, "preprocessor_config.json"))
tokenizer_class = ltxv_gemma_tokenizer(tokenizer_path, max_length=max_length)
processor = None
try:
image_processor = AutoImageProcessor.from_pretrained(
str(processor_path),
local_files_only=True,
)
processor = Gemma3Processor(
image_processor=image_processor,
tokenizer=tokenizer_class().tokenizer,
)
logger.info(f"Loaded processor from {model_root} - enhancement enabled")
except Exception as e:
logger.warning(f"Could not load processor from {model_root}: {e}")
clip_dtype = torch.bfloat16
ltxv_full_path = folder_paths.get_full_path("checkpoints", ltxv_path)
clip_target = comfy.supported_models_base.ClipTarget(
tokenizer=tokenizer_class,
clip=ltxv_gemma_clip(
gemma_model_path, ltxv_full_path, processor=processor, dtype=clip_dtype
),
)
return (comfy.sd.CLIP(clip_target),)
_UNICODE_REPLACEMENTS = str.maketrans(
"\u2018\u2019\u201c\u201d\u2014\u2013\u00a0\u2032\u2212", "''\"\"-- '-"
)
def clean_response(text):
text = text.translate(_UNICODE_REPLACEMENTS)
# Remove leading non-letter characters
for i, char in enumerate(text):
if char.isalpha():
return text[i:]
return text
@comfy_node(name="LTXVGemmaEnhancePrompt", description="Gemma 3 Prompt Enhancer")
class LTXVGemmaEnhancePrompt:
"""Enhance prompts using Gemma 3 model. Supports T2V and I2V modes."""
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"clip": ("CLIP",),
"prompt": ("STRING", {"multiline": True, "default": ""}),
"system_prompt": (
"STRING",
{
"multiline": True,
"default": DEFAULT_T2V_SYSTEM_PROMPT,
},
),
"max_tokens": (
"INT",
{"default": 512, "min": 32, "max": 1024, "step": 16},
),
"bypass_i2v": ("BOOLEAN", {"default": False}),
},
"optional": {
"image": ("IMAGE",),
"seed": (
"INT",
{"default": 42, "min": 0, "max": 0xFFFFFFFF},
),
},
}
RETURN_TYPES = ("STRING",)
RETURN_NAMES = ("enhanced_prompt",)
FUNCTION = "enhance"
CATEGORY = "lightricks/LTXV"
TITLE = "LTXV Gemma Enhance Prompt"
OUTPUT_NODE = True
DESCRIPTION = (
"Enhance text prompts using Gemma 3 VLLM for improved video generation."
)
def enhance(
self,
clip,
prompt: str,
system_prompt: str,
max_tokens: int,
bypass_i2v: bool,
image: Optional[torch.Tensor] = None,
seed: int = 42,
):
if not isinstance(seed, int):
seed = 42
clip.load_model()
encoder = clip.cond_stage_model
if encoder.processor is None:
raise ValueError(
"Processor not loaded - enhancement not available. "
"Ensure your model directory has chat_template.json, processor_config.json, "
"and preprocessor_config.json files."
)
# Determine mode: use I2V if image is provided and not bypassed
use_i2v = image is not None and not bypass_i2v
# Auto-select the appropriate system prompt if user is using default T2V prompt
if use_i2v and system_prompt.strip() == DEFAULT_T2V_SYSTEM_PROMPT.strip():
system_prompt = DEFAULT_I2V_SYSTEM_PROMPT
logger.info("Auto-selected I2V system prompt for image-to-video mode")
if not system_prompt or not system_prompt.strip():
raise ValueError(
"system_prompt is required and cannot be empty or whitespace-only"
)
if use_i2v:
pil_image = tensor_to_pil(image)
enhanced_prompt = encoder.enhance_i2v(
prompt=prompt,
image=pil_image,
system_prompt=system_prompt,
max_new_tokens=max_tokens,
seed=seed,
)
else:
enhanced_prompt = encoder.enhance_t2v(
prompt=prompt,
system_prompt=system_prompt,
max_new_tokens=max_tokens,
seed=seed,
)
enhanced_prompt = clean_response(enhanced_prompt)
return (enhanced_prompt,)
gemma_encoder_mgpu_loader.py
+297
View File
@@ -0,0 +1,297 @@
import json
import logging
import weakref
from pathlib import Path
from types import MethodType
import accelerate
import comfy.sd
import comfy.supported_models_base
import folder_paths
import safetensors
import torch
from comfy.ldm.lightricks.model import LTXFrequenciesPrecision, LTXRopeType
from comfy.utils import load_torch_file
from transformers import (
AutoImageProcessor,
Gemma3ForConditionalGeneration,
Gemma3Processor,
)
from .embeddings_connector import Embeddings1DConnector
from .gemma_encoder import (
GemmaFeaturesExtractorProjLinear,
LTXVGemmaTextEncoderModel,
find_matching_dir,
ltxv_gemma_tokenizer,
)
from .nodes_registry import comfy_node
logger = logging.getLogger(__name__)
PREFIX_BASE = "model.diffusion_model."
def _is_device_like(x):
return isinstance(x, (str, torch.device, int)) or isinstance(
x, torch.Tensor
) # to(tensor) → matches tensor's device/dtype
def _is_dtype_like(x):
# torch.float16 / torch.bfloat16 / torch.float32, etc.
return isinstance(x, torch.dtype)
def protect_module_transfers(module: torch.nn.Module, warn=False):
"""
Make .to/.cuda/.cpu NO-OPs for device moves across an entire module tree,
while still allowing dtype/memory_format conversions.
"""
def wrap_one(m: torch.nn.Module):
if hasattr(m, "__to_patched"): # idempotent
return
orig_to = m.__class__.to
proxy = weakref.proxy(m)
def safe_to(self, *args, **kwargs):
# Block explicit/implicit device moves
device_in_kwargs = "device" in kwargs
device_like_positional = len(args) > 0 and _is_device_like(args[0])
# Allow pure dtype/memory_format conversions:
# .to(torch.float16)
# .to(dtype=torch.float16)
# .to(memory_format=...)
if device_in_kwargs or device_like_positional:
if warn:
print("[protect_module_transfers] Ignoring device move .to(...).")
return self # NO-OP for device moves
# Otherwise (dtype / memory_format / non_blocking without device) → delegate
return orig_to(self, *args, **kwargs)
# Bind methods
m.to = MethodType(safe_to, proxy)
m.cuda = MethodType(lambda self, *a, **k: self, proxy) # NO-OP
m.cpu = MethodType(lambda self, *a, **k: self, proxy) # NO-OP
m.__to_patched = True
# Recurse over the whole tree
for sub in module.modules(): # includes module itself and children
wrap_one(sub)
return module
def load_proj_matrix_from_ltxv(ltxv_path: Path, prefix=""):
with safetensors.safe_open(ltxv_path, framework="pt", device="cpu") as f:
keys = filter(lambda key: key.startswith(prefix), f.keys())
sd = {k.removeprefix(prefix): f.get_tensor(k) for k in keys if k in f.keys()}
if not sd:
return None
with torch.device("meta"):
model = GemmaFeaturesExtractorProjLinear()
for name, tensor in sd.items():
accelerate.utils.set_module_tensor_to_device(model, name, "cpu", value=tensor)
device_map = accelerate.infer_auto_device_map(
model,
max_memory=None,
)
model = accelerate.dispatch_model(
model,
device_map=device_map,
)
return model
def load_proj_matrix_from_checkpoint(checkpoint_path: Path):
"""
Load model weights from a checkpoint file.
:param checkpoint_path: Path to the checkpoint file.
"""
with torch.device("meta"):
model = GemmaFeaturesExtractorProjLinear()
model = accelerate.load_checkpoint_and_dispatch(
model, str(checkpoint_path.resolve()), device_map="auto"
)
return model
def ltxv_gemma_clip(encoder_path, ltxv_path, processor=None, dtype=None):
class _LTXVGemmaTextEncoderModel(LTXVGemmaTextEncoderModel):
def __init__(self, device="cpu", dtype=dtype, model_options={}):
dtype = torch.bfloat16 # TODO: make this configurable
gemma_model = Gemma3ForConditionalGeneration.from_pretrained(
encoder_path,
local_files_only=True,
torch_dtype=dtype,
device_map="auto",
)
feature_extractor_linear = load_proj_matrix_from_ltxv(
ltxv_path, "text_embedding_projection."
)
if feature_extractor_linear is None:
feature_extractor_linear = load_proj_matrix_from_checkpoint(
encoder_path / "proj_linear.safetensors"
)
embeddings_connector = load_video_embeddings_connector(ltxv_path)
audio_embeddings_connector = load_audio_embeddings_connector(ltxv_path)
super().__init__(
model=gemma_model,
feature_extractor_linear=feature_extractor_linear,
embeddings_connector=embeddings_connector,
audio_embeddings_connector=audio_embeddings_connector,
processor=processor,
dtype=dtype,
device=device,
)
# Due to memory management issue on mgpu we need to disable the offloading of sharded gemma model.
protect_module_transfers(self)
return _LTXVGemmaTextEncoderModel
def load_video_embeddings_connector(ltxv_path, dtype=torch.bfloat16):
sd, metadata = load_torch_file(str(ltxv_path), return_metadata=True)
config = json.loads(metadata.get("config", "{}"))
transformer_config = config.get("transformer", {})
rope_type = LTXRopeType.from_dict(transformer_config)
frequencies_precision = LTXFrequenciesPrecision.from_dict(transformer_config)
pe_max_pos = transformer_config.get("connector_positional_embedding_max_pos", [1])
sd_keys = list(sd.keys())
video_only_connector_prefix = f"{PREFIX_BASE}embeddings_connector."
av_connector_prefix = f"{PREFIX_BASE}video_embeddings_connector."
prefix = (
av_connector_prefix
if f"{PREFIX_BASE}audio_adaln_single.linear.weight" in sd_keys
else video_only_connector_prefix
)
return load_embeddings_connector(
sd, prefix, dtype, rope_type, frequencies_precision, pe_max_pos
)
def load_audio_embeddings_connector(ltxv_path, dtype=torch.bfloat16):
sd, metadata = load_torch_file(str(ltxv_path), return_metadata=True)
config = json.loads(metadata.get("config", "{}"))
transformer_config = config.get("transformer", {})
rope_type = LTXRopeType.from_dict(transformer_config)
frequencies_precision = LTXFrequenciesPrecision.from_dict(transformer_config)
pe_max_pos = transformer_config.get("connector_positional_embedding_max_pos", [1])
return load_embeddings_connector(
sd,
f"{PREFIX_BASE}audio_embeddings_connector.",
dtype,
rope_type,
frequencies_precision,
pe_max_pos,
)
def load_embeddings_connector(
sd,
connector_prefix,
dtype=torch.bfloat16,
rope_type=LTXRopeType.INTERLEAVED,
frequencies_precision=LTXFrequenciesPrecision.FLOAT32,
pe_max_pos=None,
):
sd_connector = {
k[len(connector_prefix) :]: v
for k, v in sd.items()
if k.startswith(connector_prefix)
}
if len(sd_connector) == 0:
return None
operations = comfy.ops.pick_operations(dtype, dtype, disable_fast_fp8=True)
connector = Embeddings1DConnector(
dtype=dtype,
operations=operations,
positional_embedding_max_pos=pe_max_pos if pe_max_pos is not None else [1],
split_rope=rope_type == LTXRopeType.SPLIT,
double_precision_rope=frequencies_precision == LTXFrequenciesPrecision.FLOAT64,
)
connector.load_state_dict(sd_connector)
device_map = accelerate.infer_auto_device_map(connector)
connector = accelerate.dispatch_model(connector, device_map=device_map)
return connector
@comfy_node(
name="LTXVGemmaCLIPModelLoaderMGPU", description="Gemma 3 Model Loader on MGPU"
)
class LTXVGemmaCLIPModelLoaderMGPU:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"gemma_path": (
folder_paths.get_filename_list("text_encoders"),
{"tooltip": "The name of the text encoder model to load."},
),
"ltxv_path": (
folder_paths.get_filename_list("checkpoints"),
{"tooltip": "The name of the ltxv model to load."},
),
"max_length": (
"INT",
{"default": 1024, "min": 16, "max": 131072, "step": 8},
),
}
}
RETURN_TYPES = ("CLIP",)
RETURN_NAMES = ("clip",)
FUNCTION = "load_model"
CATEGORY = "lightricks/LTXV"
TITLE = "LTXV Gemma CLIP Loader"
OUTPUT_NODE = False
def load_model(self, gemma_path: str, ltxv_path: str, max_length: int):
path = Path(folder_paths.get_full_path("text_encoders", gemma_path))
model_root = path.parents[1]
tokenizer_path = Path(find_matching_dir(model_root, "tokenizer.model"))
gemma_model_path = Path(find_matching_dir(model_root, "model*.safetensors"))
tokenizer_class = ltxv_gemma_tokenizer(tokenizer_path, max_length=max_length)
processor = None
try:
image_processor = AutoImageProcessor.from_pretrained(
str(model_root),
local_files_only=True,
)
processor = Gemma3Processor(
image_processor=image_processor,
tokenizer=tokenizer_class().tokenizer,
)
logger.info(f"Loaded processor from {model_root} - enhancement enabled")
except Exception as e:
logger.warning(f"Could not load processor from {model_root}: {e}")
clip_dtype = torch.bfloat16
ltxv_full_path = folder_paths.get_full_path("checkpoints", ltxv_path)
clip_target = comfy.supported_models_base.ClipTarget(
tokenizer=tokenizer_class,
clip=ltxv_gemma_clip(
gemma_model_path, ltxv_full_path, processor=processor, dtype=clip_dtype
),
)
return (comfy.sd.CLIP(clip_target),)
getting_started.md
-68
View File
@@ -1,68 +0,0 @@
## Getting Started
Although sample nodes and a workflow can be found within ComfyUI, for best results we recommend the below process:
### Installation
1. Install ComfyUI
2. Clone this repository to `custom-nodes` folder in your ComfyUI installation directory.
3. Install the required packages:
```bash
cd custom_nodes/ComfyUI-LTXVideo && pip install -r requirements.txt
```
For portable ComfyUI installations, run
```
.\python_embeded\python.exe -m pip install -r .\ComfyUI\custom_nodes\ComfyUI-LTXVideo\requirements.txt
```
### Models
1. Download [ltx-video-2b-v0.9.1.safetensors](https://huggingface.co/Lightricks/LTX-Video/blob/main/ltx-video-2b-v0.9.1.safetensors) from Hugging Face and place it under `models/checkpoints`.
2. Install a t5 text encoder. We recommend [google_t5-v1_1-xxl_encoderonly](https://huggingface.co/mcmonkey/google_t5-v1_1-xxl_encoderonly/tree/main). You can install it using ComfyUI Model Manager.
## Example Workflows
Select a workflow from the examples and download it.
Note that to run the example workflows below, you need to have some additional custom nodes, like [ComfyUI-VideoHelperSuite](https://github.com/kosinkadink/ComfyUI-VideoHelperSuite) and others, installed. You can add them by pressing "Install Missing Custom Nodes" button in ComfyUI Manager.
### Long Video Generation
🧩 [Image to Video Long Video](example_workflows/ltxv-13b-i2v-long-multi-prompt.json): Long video generation with support for multiple prompts along the video duration.<br>
🧩 [Video to Video Long Video](example_workflows/ltxv-13b-v2v-long-depth.json): Long video-to-video generation. Given a guiding video—such as depth, pose, or edges—the flow generates a new video.
### Video Upscaling
🧩 [Video Upscaling](example_workflows/ltxv-13b-upscale.json): Upscales and adds fine details to any given video, increasing its spatial resolution by 2×.
### Easy to use multi scale generation workflows
🧩 [Image to video mixed](example_workflows/ltxv13b-i2v-mixed-multiscale.json): mixed flow with full and distilled model for best quality and speed trade-off.<br>
### 13B model<br>
🧩 [Image to video](example_workflows/ltxv-13b-i2v-base.json)<br>
🧩 [Image to video with keyframes](example_workflows/ltxv-13b-i2v-keyframes.json)<br>
🧩 [Image to video with duration extension](example_workflows/ltxv-13b-i2v-extend.json)<br>
🧩 [Image to video 8b quantized](example_workflows/ltxv-13b-i2v-base-fp8.json)
### 13B distilled model<br>
🧩 [Image to video](example_workflows/13b-distilled/ltxv-13b-dist-i2v-base.json)<br>
🧩 [Image to video with keyframes](example_workflows/13b-distilled/ltxv-13b-dist-i2v-keyframes.json)<br>
🧩 [Image to video with duration extension](example_workflows/13b-distilled/ltxv-13b-dist-i2v-extend.json)<br>
🧩 [Image to video 8b quantized](example_workflows/13b-distilled/ltxv-13b-dist-i2v-base-fp8.json)
### ICLora
🧩 [Download workflow](example_workflows/ic_lora/ic-lora.json)
### Inversion
#### Flow Edit
🧩 [Download workflow](example_workflows/tricks/ltxvideo-flow-edit.json)<br>
![workflow](example_workflows/tricks/ltxvideo-flow-edit.png)
#### RF Edit
🧩 [Download workflow](example_workflows/tricks/ltxvideo-rf-edit.json)<br>
![workflow](example_workflows/tricks/ltxvideo-rf-edit.png)
guide.py
+5 -5
View File
@@ -13,7 +13,7 @@ def blur_internal(image, blur_radius):
# relationship between sigma and kernel size 2*blur_radius + 1, however we want somewhat weaker
# blurring, so we use 0.3 * blur_radius instead, reducing the sigma value by 0.5
sigma = 0.3 * blur_radius
image = post_processing.Blur().blur(image, blur_radius, sigma)[0]
image = post_processing.Blur.execute(image, blur_radius, sigma)[0]
return image
@@ -124,9 +124,9 @@ class LTXVAddGuideAdvanced:
.movedim(1, -1)
.clamp(0, 1)
)
image = nodes_lt.LTXVPreprocess().preprocess(image, crf)[0]
image = nodes_lt.LTXVPreprocess().execute(image, crf)[0]
image = blur_internal(image, blur_radius)
return nodes_lt.LTXVAddGuide().generate(
return nodes_lt.LTXVAddGuide().execute(
positive=positive,
negative=negative,
vae=vae,
@@ -239,9 +239,9 @@ class LTXVImgToVideoAdvanced:
image = comfy.utils.common_upscale(
image.movedim(-1, 1), width, height, interpolation, crop=crop
).movedim(1, -1)
image = nodes_lt.LTXVPreprocess().preprocess(image, crf)[0]
image = nodes_lt.LTXVPreprocess().execute(image, crf)[0]
image = blur_internal(image, blur_radius)
return nodes_lt.LTXVImgToVideo().generate(
return nodes_lt.LTXVImgToVideo().execute(
positive=positive,
negative=negative,
vae=vae,
guiders/__init__.py
+2
View File
@@ -0,0 +1,2 @@
from .multimodal_guider import MultimodalGuiderNode # noqa: F401
from .parameters import GuiderParametersNode # noqa: F401
guiders/multimodal_guider.py
+323
View File
@@ -0,0 +1,323 @@
from typing import List
import comfy.samplers
import torch
from comfy.model_patcher import ModelPatcher
from comfy.patcher_extension import CallbacksMP
from ..nodes_registry import comfy_node
from ..stg import STGBlockWrapper, STGFlag
from .parameters import GuiderParameters, Modality
class MultimodalGuider(comfy.samplers.CFGGuider):
def __init__(
self,
model: ModelPatcher,
parameters: dict[str, GuiderParameters],
skip_blocks: List[int],
):
model = model.clone()
self.current_step = 0
self.last_denoised_v = None
self.last_denoised_a = None
model.add_callback_with_key(
CallbacksMP.ON_PRE_RUN, "mm_guider_on_pre_run", self.reset_current_step
)
super().__init__(model)
self.stg_flag = STGFlag(
do_skip=False,
skip_layers=skip_blocks,
)
self.patch_model(model, self.stg_flag)
self.parameters = parameters
def reset_current_step(self, model_patcher=None):
self.current_step = 0
self.last_denoised_v = None
self.last_denoised_a = None
@classmethod
def patch_model(cls, model: ModelPatcher, stg_flag: STGFlag):
transformer_blocks = cls.get_transformer_blocks(model)
for i, block in enumerate(transformer_blocks):
model.set_model_patch_replace(
STGBlockWrapper(block, stg_flag, i), "dit", "double_block", i
)
@staticmethod
def get_transformer_blocks(model: ModelPatcher):
diffusion_model = model.get_model_object("diffusion_model")
key = "diffusion_model.transformer_blocks"
if diffusion_model.__class__.__name__ == "LTXVTransformer3D":
key = "diffusion_model.transformer.transformer_blocks"
return model.get_model_object(key)
def set_conds(self, positive, negative):
self.inner_set_conds({"positive": positive, "negative": negative})
def calc_stg_indexes(
self,
run_vx: bool,
run_ax: bool,
audio_length: int,
audio_ptb: bool,
video_ptb: bool,
):
stg_indexes = set()
num_self_attns = 0
if run_vx:
num_self_attns += 1
if run_ax and audio_length > 0:
num_self_attns += 1
video_attn_idx = 0
audio_attn_idx = 0 if num_self_attns == 1 else 2
if video_ptb:
stg_indexes.add(video_attn_idx)
if audio_ptb:
stg_indexes.add(audio_attn_idx)
return list(stg_indexes)
def predict_noise(
self,
x: torch.Tensor,
timestep: torch.Tensor,
model_options: dict = {},
seed=None,
):
x_shape = x.shape
# in CFGGuider.predict_noise, we call sampling_function(), which uses cfg_function() to compute pos & neg
# but we'd rather do a single batch of sampling pos, neg, and perturbed, so we call calc_cond_batch([perturbed,pos,neg]) directly
current_step = self.current_step
self.current_step = current_step + 1
positive_cond = self.conds.get("positive", None)
negative_cond = self.conds.get("negative", None)
audio_params: GuiderParameters = self.parameters.get(
Modality.AUDIO.value, GuiderParameters()
)
video_params: GuiderParameters = self.parameters.get(
Modality.VIDEO.value, GuiderParameters()
)
audio_length = model_options["transformer_options"].get("audio_length", 0)
run_vx = not video_params.do_skip(current_step)
run_ax = not audio_params.do_skip(current_step)
run_a2v = video_params.do_cross_attn(current_step)
run_v2a = audio_params.do_cross_attn(current_step)
vx, ax = self.inner_model.diffusion_model.separate_audio_and_video_latents(
x, audio_length
)
if not run_vx:
vx = self.last_denoised_v
if not run_ax:
ax = self.last_denoised_a
x = self.inner_model.diffusion_model.recombine_audio_and_video_latents(
vx, ax, x_shape
)
if not run_vx and not run_ax:
return x
try:
model_options["transformer_options"]["run_vx"] = run_vx
model_options["transformer_options"]["run_ax"] = run_ax
model_options["transformer_options"]["a2v_cross_attn"] = run_a2v
model_options["transformer_options"]["v2a_cross_attn"] = run_v2a
noise_pred_pos = comfy.samplers.calc_cond_batch(
self.inner_model,
[positive_cond],
x,
timestep,
model_options,
)[0]
finally:
del model_options["transformer_options"]["run_vx"]
del model_options["transformer_options"]["run_ax"]
del model_options["transformer_options"]["a2v_cross_attn"]
del model_options["transformer_options"]["v2a_cross_attn"]
v_noise_pred_pos, a_noise_pred_pos = (
self.inner_model.diffusion_model.separate_audio_and_video_latents(
noise_pred_pos, audio_length
)
)
a_noise_pred_neg, v_noise_pred_neg = 0, 0
a_noise_pred_perturbed, v_noise_pred_perturbed = 0, 0
a_noise_pred_modality, v_noise_pred_modality = 0, 0
if any(params.do_uncond() for params in [audio_params, video_params]):
try:
model_options["transformer_options"]["run_vx"] = run_vx
model_options["transformer_options"]["run_ax"] = run_ax
model_options["transformer_options"]["a2v_cross_attn"] = run_a2v
model_options["transformer_options"]["v2a_cross_attn"] = run_v2a
noise_pred_neg = comfy.samplers.calc_cond_batch(
self.inner_model,
[negative_cond],
x,
timestep,
model_options,
)[0]
v_noise_pred_neg, a_noise_pred_neg = (
self.inner_model.diffusion_model.separate_audio_and_video_latents(
noise_pred_neg, audio_length
)
)
finally:
del model_options["transformer_options"]["run_vx"]
del model_options["transformer_options"]["run_ax"]
del model_options["transformer_options"]["a2v_cross_attn"]
del model_options["transformer_options"]["v2a_cross_attn"]
if any(params.do_perturbed() for params in [audio_params, video_params]):
try:
stg_indexes = self.calc_stg_indexes(
run_vx,
run_ax,
audio_length,
audio_params.perturb_attn,
video_params.perturb_attn,
)
model_options["transformer_options"]["run_vx"] = run_vx
model_options["transformer_options"]["run_ax"] = run_ax
model_options["transformer_options"]["a2v_cross_attn"] = run_a2v
model_options["transformer_options"]["v2a_cross_attn"] = run_v2a
model_options["transformer_options"]["ptb_index"] = 0
model_options["transformer_options"]["stg_indexes"] = stg_indexes
self.stg_flag.do_skip = True
noise_pred_perturbed = comfy.samplers.calc_cond_batch(
self.inner_model,
[positive_cond],
x,
timestep,
model_options,
)[0]
v_noise_pred_perturbed, a_noise_pred_perturbed = (
self.inner_model.diffusion_model.separate_audio_and_video_latents(
noise_pred_perturbed, audio_length
)
)
finally:
self.stg_flag.do_skip = False
del model_options["transformer_options"]["ptb_index"]
del model_options["transformer_options"]["run_vx"]
del model_options["transformer_options"]["run_ax"]
del model_options["transformer_options"]["a2v_cross_attn"]
del model_options["transformer_options"]["v2a_cross_attn"]
del model_options["transformer_options"]["stg_indexes"]
if any(params.do_modality() for params in [audio_params, video_params]):
try:
model_options["transformer_options"]["run_vx"] = run_vx
model_options["transformer_options"]["run_ax"] = run_ax
model_options["transformer_options"]["a2v_cross_attn"] = False
model_options["transformer_options"]["v2a_cross_attn"] = False
noise_pred_modality = comfy.samplers.calc_cond_batch(
self.inner_model,
[positive_cond],
x,
timestep,
model_options,
)[0]
v_noise_pred_modality, a_noise_pred_modality = (
self.inner_model.diffusion_model.separate_audio_and_video_latents(
noise_pred_modality, audio_length
)
)
finally:
del model_options["transformer_options"]["a2v_cross_attn"]
del model_options["transformer_options"]["v2a_cross_attn"]
del model_options["transformer_options"]["run_vx"]
del model_options["transformer_options"]["run_ax"]
if run_vx:
vx = video_params.calculate(
v_noise_pred_pos,
v_noise_pred_neg,
v_noise_pred_perturbed,
v_noise_pred_modality,
)
else:
vx = self.last_denoised_v
if run_ax:
ax = audio_params.calculate(
a_noise_pred_pos,
a_noise_pred_neg,
a_noise_pred_perturbed,
a_noise_pred_modality,
)
else:
ax = self.last_denoised_a
x = self.inner_model.diffusion_model.recombine_audio_and_video_latents(
vx, ax, x_shape
)
# normally this would be done in cfg_function, but we skipped
# that for efficiency: we can compute the noise predictions in
# a single call to calc_cond_batch() (rather than two)
# so we replicate the hook here
for fn in model_options.get("sampler_post_cfg_function", []):
args = {
"denoised": x,
"cond": positive_cond,
"uncond": negative_cond,
"model": self.inner_model,
"uncond_denoised": noise_pred_neg,
"cond_denoised": noise_pred_pos,
"sigma": timestep,
"model_options": model_options,
"input": x,
# not in the original call in samplers.py:cfg_function, but made available for future hooks
"perturbed_cond": positive_cond,
"perturbed_cond_denoised": noise_pred_perturbed,
}
x = fn(args)
self.last_denoised_v, self.last_denoised_a = (
self.inner_model.diffusion_model.separate_audio_and_video_latents(
x, audio_length
)
)
return x
@comfy_node(name="MultimodalGuider")
class MultimodalGuiderNode:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"model": ("MODEL",),
"positive": ("CONDITIONING",),
"negative": ("CONDITIONING",),
"parameters": ("GUIDER_PARAMETERS",),
"skip_blocks": (
"STRING",
{"default": "", "multiline": True},
),
}
}
RETURN_TYPES = ("GUIDER",)
FUNCTION = "get_guider"
CATEGORY = "lightricks/LTXV"
def get_guider(self, model, positive, negative, parameters, skip_blocks):
skip_blocks = [int(n.strip()) for n in skip_blocks.split(",") if n.strip()]
guider = MultimodalGuider(model, parameters, skip_blocks)
guider.set_conds(positive, negative)
guider.raw_conds = (positive, negative)
return (guider,)
guiders/parameters.py
+164
View File
@@ -0,0 +1,164 @@
import math
from enum import Enum
from ..nodes_registry import comfy_node
class Modality(Enum):
VIDEO = "VIDEO"
AUDIO = "AUDIO"
class GuiderParameters:
def __init__(
self,
cfg_scale: float = 1.0,
stg_scale: float = 0.0,
perturb_attn: bool = True,
rescale_scale: float = 0.0,
modality_scale: float = 1.0,
skip_step: int = 0,
cross_attn: bool = True,
):
self.cfg_scale = cfg_scale
self.stg_scale = stg_scale
self.perturb_attn = perturb_attn
self.rescale_scale = rescale_scale
self.modality_scale = modality_scale
self.skip_step = skip_step
self.cross_attn = cross_attn
def __str__(self):
return f"cfg_scale: {self.cfg_scale}, stg_scale: {self.stg_scale}, rescale_scale: {self.rescale_scale}, modality_scale: {self.modality_scale}"
def __repr__(self):
return f"cfg_scale: {self.cfg_scale}, stg_scale: {self.stg_scale}, rescale_scale: {self.rescale_scale}, modality_scale: {self.modality_scale}"
def calculate(
self, noise_pred_pos, noise_pred_neg, noise_pred_pertubed, noise_pred_modality
):
noise_pred = (
noise_pred_pos
+ (self.cfg_scale - 1) * (noise_pred_pos - noise_pred_neg)
+ self.stg_scale * (noise_pred_pos - noise_pred_pertubed)
+ (self.modality_scale - 1) * (noise_pred_pos - noise_pred_modality)
)
if self.rescale_scale != 0:
factor = noise_pred_pos.std() / noise_pred.std()
factor = self.rescale_scale * factor + (1 - self.rescale_scale)
noise_pred = noise_pred * factor
return noise_pred
def do_uncond(self):
return not math.isclose(self.cfg_scale, 1.0)
def do_perturbed(self):
return not math.isclose(self.stg_scale, 0.0)
def do_modality(self):
return not math.isclose(self.modality_scale, 1.0)
def do_skip(self, step: int) -> bool:
if self.skip_step == 0:
return False
return step % (self.skip_step + 1) != 0
def do_cross_attn(self, step: int) -> bool:
return self.cross_attn and not self.do_skip(step)
@comfy_node(name="GuiderParameters")
class GuiderParametersNode:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"modality": (
[m.value for m in Modality],
{
"default": Modality.VIDEO.value,
},
),
"cfg": (
"FLOAT",
{
"default": 1.0,
"min": 0.0,
"max": 100.0,
"step": 0.1,
"round": 0.01,
},
),
"stg": (
"FLOAT",
{"default": 1.0, "min": 0.0, "max": 100.0, "step": 0.01},
),
"perturb_attn": (
"BOOLEAN",
{"default": True},
),
"rescale": (
"FLOAT",
{"default": 0.7, "min": 0.0, "max": 100.0, "step": 0.01},
),
"modality_scale": (
"FLOAT",
{"default": 0.0, "min": 0.0, "max": 100.0, "step": 0.01},
),
"skip_step": (
"INT",
{"default": 0, "min": 0, "max": 100, "step": 1},
),
"cross_attn": (
"BOOLEAN",
{"default": True},
),
},
"optional": {
"parameters": (
"GUIDER_PARAMETERS",
{"default": None},
),
},
}
RETURN_TYPES = ("GUIDER_PARAMETERS",)
FUNCTION = "get_parameters"
CATEGORY = "lightricks/LTXV"
def get_parameters(
self,
modality,
cfg,
stg,
perturb_attn,
rescale,
modality_scale,
skip_step,
cross_attn,
parameters=None,
):
parameters = parameters.copy() if parameters is not None else {}
if modality in parameters:
raise ValueError(f"Modality {modality} already exists in parameters")
parameters.update(
{
modality: GuiderParameters(
cfg,
stg,
perturb_attn,
rescale,
modality_scale,
skip_step,
cross_attn,
)
}
)
return (parameters,)
latent_adain.py
-51
View File
@@ -1,51 +0,0 @@
import copy
import torch
from .nodes_registry import comfy_node
@comfy_node(name="LTXVAdainLatent")
class LTXVAdainLatent:
def __init__(self):
pass
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"latents": ("LATENT",),
"reference": ("LATENT",),
"factor": (
"FLOAT",
{
"default": 1.0,
"min": -10.0,
"max": 10.0,
"step": 0.01,
"round": 0.01,
},
),
},
}
RETURN_TYPES = ("LATENT",)
FUNCTION = "batch_normalize"
CATEGORY = "Lightricks/latents"
def batch_normalize(self, latents, reference, factor):
latents_copy = copy.deepcopy(latents)
t = latents_copy["samples"] # B x C x F x H x W
for i in range(t.size(0)): # batch
for c in range(t.size(1)): # channel
r_sd, r_mean = torch.std_mean(
reference["samples"][i, c], dim=None
) # index by original dim order
i_sd, i_mean = torch.std_mean(t[i, c], dim=None)
t[i, c] = ((t[i, c] - i_mean) / i_sd) * r_sd + r_mean
latents_copy["samples"] = torch.lerp(latents["samples"], t, factor)
return (latents_copy,)
latent_norm.py
+331
View File
@@ -0,0 +1,331 @@
import copy
from typing import Callable
import torch
from comfy.model_patcher import ModelPatcher
from .nodes_registry import comfy_node
@comfy_node(name="LTXVAdainLatent")
class LTXVAdainLatent:
def __init__(self):
pass
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"latents": ("LATENT",),
"reference": ("LATENT",),
"factor": (
"FLOAT",
{
"default": 1.0,
"min": -10.0,
"max": 10.0,
"step": 0.01,
"round": 0.01,
},
),
},
"optional": {
"per_frame": ("BOOLEAN", {"default": False}),
},
}
RETURN_TYPES = ("LATENT",)
FUNCTION = "batch_normalize"
CATEGORY = "Lightricks/latents"
def batch_normalize(self, latents, reference, factor, per_frame=False):
latents_copy = copy.deepcopy(latents)
t = latents_copy["samples"] # B x C x F x H x W
if per_frame:
if reference["samples"].size(2) == 1:
print("Reference has only one frame, using it for all frames")
reference["samples"] = reference["samples"].repeat(
1, 1, t.size(2), 1, 1
)
elif t.size(2) > reference["samples"].size(2):
raise ValueError("Latents have more frames than reference")
for i in range(t.size(0)): # batch
for c in range(t.size(1)):
if not per_frame:
r_sd, r_mean = torch.std_mean(
reference["samples"][i, c], dim=None
) # index by original dim order
i_sd, i_mean = torch.std_mean(t[i, c], dim=None)
t[i, c] = ((t[i, c] - i_mean) / i_sd) * r_sd + r_mean
else:
for f in range(t.size(2)):
r_sd, r_mean = torch.std_mean(
reference["samples"][i, c, f], dim=None
) # index by original dim order
i_sd, i_mean = torch.std_mean(t[i, c, f], dim=None)
t[i, c, f] = ((t[i, c, f] - i_mean) / i_sd) * r_sd + r_mean
latents_copy["samples"] = torch.lerp(latents["samples"], t, factor)
return (latents_copy,)
@comfy_node(name="LTXVStatNormLatent")
class LTXVStatNormLatent:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"latents": ("LATENT",),
"target_mean": (
"FLOAT",
{
"default": 0.0,
"min": -10.0,
"max": 10.0,
"step": 0.01,
"round": 0.01,
},
),
"target_std": (
"FLOAT",
{
"default": 1.0,
"min": 0.01,
"max": 10.0,
"step": 0.01,
"round": 0.01,
},
),
"percentile": (
"FLOAT",
{
"default": 95.0,
"min": 50.0,
"max": 100.0,
"step": 0.1,
"round": 0.1,
"tooltip": "Percentile of distribution to use for statistics calculation",
},
),
"factor": (
"FLOAT",
{
"default": 1.0,
"min": -10.0,
"max": 10.0,
"step": 0.01,
"round": 0.01,
},
),
"clip_outliers": ("BOOLEAN", {"default": False}),
},
}
RETURN_TYPES = ("LATENT",)
FUNCTION = "statistical_normalize"
CATEGORY = "Lightricks/latents"
def statistical_normalize(
self, latents, target_mean, target_std, percentile, factor, clip_outliers
):
latents_copy = copy.deepcopy(latents)
t = latents_copy["samples"] # B x C x F x H x W
# For 95% of distribution, we want to exclude 2.5% from each tail
lower_percentile = (100 - percentile) / 2
upper_percentile = 100 - lower_percentile
for i in range(t.size(0)): # batch
for c in range(t.size(1)): # channel
channel_data = t[i, c]
original_shape = channel_data.shape
channel_flat = channel_data.flatten()
# Calculate percentiles
lower_bound = torch.quantile(channel_flat, lower_percentile / 100)
upper_bound = torch.quantile(channel_flat, upper_percentile / 100)
# Create mask for values within the percentile range
mask = (channel_flat >= lower_bound) & (channel_flat <= upper_bound)
# Calculate mean and std only on the masked values
if mask.sum() > 0:
filtered_data = channel_flat[mask]
current_mean = filtered_data.mean()
current_std = filtered_data.std()
# Avoid division by zero
if current_std > 1e-8:
# Normalize all values
normalized_flat = (
(channel_flat - current_mean) / current_std
) * target_std + target_mean
if clip_outliers:
# Calculate the normalized bounds
normalized_lower = (
(lower_bound - current_mean) / current_std
) * target_std + target_mean
normalized_upper = (
(upper_bound - current_mean) / current_std
) * target_std + target_mean
# Clip outliers to the normalized bounds
normalized_flat = torch.where(
channel_flat < lower_bound,
normalized_lower,
normalized_flat,
)
normalized_flat = torch.where(
channel_flat > upper_bound,
normalized_upper,
normalized_flat,
)
# Reshape back to original shape
t[i, c] = normalized_flat.reshape(original_shape)
else:
# If std is too small, just shift by mean
t[i, c] = channel_data - current_mean + target_mean
latents_copy["samples"] = torch.lerp(latents["samples"], t, factor)
return (latents_copy,)
class PerStepNormPatcher:
"""
Base class for per-step normalization nodes.
"""
@classmethod
def required(s):
return {
"model": ("MODEL",),
"factors": (
"STRING",
{
"default": "0.9, 0.75, 0.0",
"tooltip": "Comma-separated list of factors, each factor will be used for one step.",
},
),
}
@classmethod
def optional(s):
return {}
@staticmethod
def patch_smooth_norm(
model: ModelPatcher, factors: str, normalize_fn: Callable
) -> ModelPatcher:
model = model.clone()
factors = [float(x) for x in factors.split(",")]
step = 0
def norm_fn(args):
nonlocal step
latent = args["denoised"]
factor = factors[min(step, len(factors) - 1)]
step += 1
out_latent = normalize_fn(latent={"samples": latent}, factor=factor)
return out_latent[0]["samples"]
model.set_model_sampler_post_cfg_function(norm_fn)
return (model,)
@classmethod
def IS_CHANGED(s, *args, **kwargs):
return float("NaN")
@comfy_node(name="LTXVPerStepAdainPatcher")
class LTXVPerStepAdainPatcher(PerStepNormPatcher):
@classmethod
def INPUT_TYPES(s):
return {
"required": super().required()
| {
"reference": ("LATENT",),
},
"optional": super().optional()
| {
"per_frame": ("BOOLEAN", {"default": False}),
},
}
RETURN_TYPES = ("MODEL",)
FUNCTION = "patch_model"
CATEGORY = "Lightricks/latents"
def patch_model(self, model, factors, reference, per_frame=False):
def cfg_adain(latent, factor):
return LTXVAdainLatent().batch_normalize(
latent, reference, factor, per_frame
)
return self.patch_smooth_norm(model, factors, cfg_adain)
@comfy_node(name="LTXVPerStepStatNormPatcher")
class LTXVPerStepStatNormPatcher(PerStepNormPatcher):
@classmethod
def INPUT_TYPES(s):
return {
"required": super().required()
| {
"target_mean": (
"FLOAT",
{
"default": 0.0,
"min": -10.0,
"max": 10.0,
"step": 0.01,
"round": 0.01,
},
),
"target_std": (
"FLOAT",
{
"default": 1.0,
"min": 0.01,
"max": 10.0,
"step": 0.01,
"round": 0.01,
},
),
"percentile": (
"FLOAT",
{
"default": 95.0,
"min": 50.0,
"max": 100.0,
"step": 0.1,
"round": 0.1,
"tooltip": "Percentile of distribution to use for statistics calculation",
},
),
"clip_outliers": ("BOOLEAN", {"default": False}),
},
}
RETURN_TYPES = ("MODEL",)
FUNCTION = "patch_model"
CATEGORY = "Lightricks/latents"
def patch_model(
self, model, factors, target_mean, target_std, percentile, clip_outliers
):
def cfg_stat_norm(latent, factor):
return LTXVStatNormLatent().statistical_normalize(
latent, target_mean, target_std, percentile, factor, clip_outliers
)
return self.patch_smooth_norm(model, factors, cfg_stat_norm)
latent_upsampler.py
-337
View File
@@ -1,337 +0,0 @@
from typing import Optional
import folder_paths
import torch
import torch.nn as nn
from comfy import model_management
from diffusers import ConfigMixin, ModelMixin
from einops import rearrange
from .nodes_registry import comfy_node
class PixelShuffle3D(nn.Module):
def __init__(self, upscale_factor: int = 2):
super().__init__()
self.r = upscale_factor
def forward(self, x):
b, c, f, h, w = x.shape
r = self.r
out_c = c // (r**3)
x = x.view(b, out_c, r, r, r, f, h, w)
x = x.permute(0, 1, 5, 2, 6, 3, 7, 4) # (b, out_c, f, r, h, r, w, r)
x = x.reshape(b, out_c, f * r, h * r, w * r)
return x
class PixelShuffle2D(nn.Module):
def __init__(self, upscale_factor: int = 2):
super().__init__()
self.r = upscale_factor
def forward(self, x):
b, c, h, w = x.size()
r = self.r
out_c = c // (r * r)
x = x.view(b, out_c, r, r, h, w)
x = x.permute(0, 1, 4, 2, 5, 3) # (b, out_c, h, r, w, r)
x = x.reshape(b, out_c, h * r, w * r)
return x
class PixelShuffle1D(nn.Module):
def __init__(self, upscale_factor: int = 2):
super().__init__()
self.r = upscale_factor
def forward(self, x):
b, c, f, h, w = x.shape
r = self.r
out_c = c // r
x = x.view(b, out_c, r, f, h, w) # [B, C//r, r, F, H, W]
x = x.permute(0, 1, 3, 2, 4, 5) # [B, C//r, F, r, H, W]
x = x.reshape(b, out_c, f * r, h, w)
return x
class ResBlock(nn.Module):
def __init__(
self, channels: int, mid_channels: Optional[int] = None, dims: int = 3
):
super().__init__()
if mid_channels is None:
mid_channels = channels
Conv = nn.Conv2d if dims == 2 else nn.Conv3d
self.conv1 = Conv(channels, mid_channels, kernel_size=3, padding=1)
self.norm1 = nn.GroupNorm(32, mid_channels)
self.conv2 = Conv(mid_channels, channels, kernel_size=3, padding=1)
self.norm2 = nn.GroupNorm(32, channels)
self.activation = nn.SiLU()
def forward(self, x: torch.Tensor) -> torch.Tensor:
residual = x
x = self.conv1(x)
x = self.norm1(x)
x = self.activation(x)
x = self.conv2(x)
x = self.norm2(x)
x = self.activation(x + residual)
return x
class LatentUpsampler(ModelMixin, ConfigMixin):
"""
Model to spatially upsample VAE latents.
Args:
in_channels (`int`): Number of channels in the input latent
mid_channels (`int`): Number of channels in the middle layers
num_blocks_per_stage (`int`): Number of ResBlocks to use in each stage (pre/post upsampling)
dims (`int`): Number of dimensions for convolutions (2 or 3)
spatial_upsample (`bool`): Whether to spatially upsample the latent
temporal_upsample (`bool`): Whether to temporally upsample the latent
"""
def __init__(
self,
in_channels: int = 128,
mid_channels: int = 512,
num_blocks_per_stage: int = 4,
dims: int = 3,
spatial_upsample: bool = True,
temporal_upsample: bool = False,
):
super().__init__()
self.in_channels = in_channels
self.mid_channels = mid_channels
self.num_blocks_per_stage = num_blocks_per_stage
self.dims = dims
self.spatial_upsample = spatial_upsample
self.temporal_upsample = temporal_upsample
Conv = nn.Conv2d if dims == 2 else nn.Conv3d
self.initial_conv = Conv(in_channels, mid_channels, kernel_size=3, padding=1)
self.initial_norm = nn.GroupNorm(32, mid_channels)
self.initial_activation = nn.SiLU()
self.res_blocks = nn.ModuleList(
[ResBlock(mid_channels, dims=dims) for _ in range(num_blocks_per_stage)]
)
if spatial_upsample and temporal_upsample:
self.upsampler = nn.Sequential(
nn.Conv3d(mid_channels, 8 * mid_channels, kernel_size=3, padding=1),
PixelShuffle3D(2),
)
elif spatial_upsample:
self.upsampler = nn.Sequential(
nn.Conv2d(mid_channels, 4 * mid_channels, kernel_size=3, padding=1),
PixelShuffle2D(2),
)
elif temporal_upsample:
self.upsampler = nn.Sequential(
nn.Conv3d(mid_channels, 2 * mid_channels, kernel_size=3, padding=1),
PixelShuffle1D(2),
)
else:
raise ValueError(
"Either spatial_upsample or temporal_upsample must be True"
)
self.post_upsample_res_blocks = nn.ModuleList(
[ResBlock(mid_channels, dims=dims) for _ in range(num_blocks_per_stage)]
)
self.final_conv = Conv(mid_channels, in_channels, kernel_size=3, padding=1)
def forward(self, latent: torch.Tensor) -> torch.Tensor:
b, c, f, h, w = latent.shape
if self.dims == 2:
x = rearrange(latent, "b c f h w -> (b f) c h w")
x = self.initial_conv(x)
x = self.initial_norm(x)
x = self.initial_activation(x)
for block in self.res_blocks:
x = block(x)
x = self.upsampler(x)
for block in self.post_upsample_res_blocks:
x = block(x)
x = self.final_conv(x)
x = rearrange(x, "(b f) c h w -> b c f h w", b=b, f=f)
else:
x = self.initial_conv(latent)
x = self.initial_norm(x)
x = self.initial_activation(x)
for block in self.res_blocks:
x = block(x)
if self.temporal_upsample:
x = self.upsampler(x)
x = x[:, :, 1:, :, :]
else:
x = rearrange(x, "b c f h w -> (b f) c h w")
x = self.upsampler(x)
x = rearrange(x, "(b f) c h w -> b c f h w", b=b, f=f)
for block in self.post_upsample_res_blocks:
x = block(x)
x = self.final_conv(x)
return x
@classmethod
def from_config(cls, config):
return cls(
in_channels=config.get("in_channels", 4),
mid_channels=config.get("mid_channels", 128),
num_blocks_per_stage=config.get("num_blocks_per_stage", 4),
dims=config.get("dims", 2),
spatial_upsample=config.get("spatial_upsample", True),
temporal_upsample=config.get("temporal_upsample", False),
)
def config(self):
return {
"_class_name": "LatentUpsampler",
"in_channels": self.in_channels,
"mid_channels": self.mid_channels,
"num_blocks_per_stage": self.num_blocks_per_stage,
"dims": self.dims,
"spatial_upsample": self.spatial_upsample,
"temporal_upsample": self.temporal_upsample,
}
def load_weights(self, weights_path: str) -> None:
"""
Load model weights from a .safetensors file and switch to evaluation mode.
Args:
weights_path (str): Path to the .safetensors file containing the model weights
Raises:
RuntimeError: If there are missing or unexpected keys in the state dict
"""
import safetensors.torch
sd = safetensors.torch.load_file(weights_path)
self.load_state_dict(sd, strict=False, assign=True)
# Switch to evaluation mode
self.eval()
@comfy_node(name="LTXVLatentUpsampler")
class LTXVLatentUpsampler:
"""
Upsamples a video latent by a factor of 2.
"""
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"samples": ("LATENT",),
"upscale_model": ("UPSCALE_MODEL",),
"vae": ("VAE",),
}
}
RETURN_TYPES = ("LATENT",)
FUNCTION = "upsample_latent"
CATEGORY = "latent/video"
def upsample_latent(
self, samples: dict, upscale_model: LatentUpsampler, vae
) -> tuple:
"""
Upsample the input latent using the provided model.
Args:
samples (dict): Input latent samples
upscale_model (LatentUpsampler): Loaded upscale model
Returns:
tuple: Tuple containing the upsampled latent
"""
latents = samples["samples"]
# Ensure latents are on the same device as the model
if latents.device != upscale_model.device:
latents = latents.to(upscale_model.device)
latents = vae.first_stage_model.per_channel_statistics.un_normalize(latents)
upsampled_latents = upscale_model(latents)
upsampled_latents = vae.first_stage_model.per_channel_statistics.normalize(
upsampled_latents
)
upsampled_latents = upsampled_latents.to(model_management.intermediate_device())
return_dict = samples.copy()
return_dict["samples"] = upsampled_latents
return (return_dict,)
@comfy_node(name="LTXVLatentUpsamplerModelLoader")
class LTXVLatentUpsamplerModelLoader:
"""
Loads a latent upsampler model from a .safetensors file.
"""
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"upscale_model": (folder_paths.get_filename_list("upscale_models"),),
"spatial_upsample": ("BOOLEAN", {"default": True}),
"temporal_upsample": ("BOOLEAN", {"default": False}),
}
}
RETURN_TYPES = ("UPSCALE_MODEL",)
FUNCTION = "load_model"
CATEGORY = "latent/video"
def load_model(
self, upscale_model: str, spatial_upsample: bool, temporal_upsample: bool
) -> tuple:
"""
Load the upscale model from the specified file.
Args:
upscale_model (str): Name of the upscale model file
Returns:
tuple: Tuple containing the loaded model
"""
upscale_model_path = folder_paths.get_full_path("upscale_models", upscale_model)
if upscale_model_path is None:
raise ValueError(f"Upscale model {upscale_model} not found")
try:
latent_upsampler = LatentUpsampler(
num_blocks_per_stage=4,
dims=3,
spatial_upsample=spatial_upsample,
temporal_upsample=temporal_upsample,
)
latent_upsampler.load_weights(upscale_model_path)
except Exception as e:
raise ValueError(
f"Failed to initialize LatentUpsampler with this configuration: {str(e)}"
)
latent_upsampler.eval()
# Move model to appropriate device
device = model_management.get_torch_device()
latent_upsampler.to(device)
return (latent_upsampler,)
latents.py
+476 -12
View File
@@ -1,7 +1,11 @@
from typing import Optional
import comfy.utils
import comfy_extras.nodes_lt as nodes_lt
import numpy as np
import torch
from comfy.ldm.lightricks.vae.audio_vae import LATENT_DOWNSAMPLE_FACTOR
from comfy.nested_tensor import NestedTensor
from .nodes_registry import comfy_node
@@ -76,7 +80,7 @@ class LTXVSelectLatents:
s["samples"] = video_latent[:, :, start_idx : end_idx + 1, :, :]
# Handle noise mask if present
if "noise_mask" in s:
if "noise_mask" in s and s["noise_mask"] is not None:
s["noise_mask"] = s["noise_mask"][:, :, start_idx : end_idx + 1, :, :]
return (s,)
@@ -160,12 +164,29 @@ class LTXVAddLatents:
def _validate_dimensions(self, latent1: torch.Tensor, latent2: torch.Tensor):
"""Validates that latent dimensions match except for frames."""
b1, c1, f1, h1, w1 = latent1.shape
b2, c2, f2, h2, w2 = latent2.shape
# video latents
if latent1.ndim == 5 and latent2.ndim == 5:
b1, c1, f1, h1, w1 = latent1.shape
b2, c2, f2, h2, w2 = latent2.shape
if not (b1 == b2 and c1 == c2 and h1 == h2 and w1 == w2):
if not (b1 == b2 and c1 == c2 and h1 == h2 and w1 == w2):
raise ValueError(
f"Latent dimensions must match (except frames dimension).\n"
f"Got shapes {latent1.shape} and {latent2.shape}"
)
# audio latents
elif latent1.ndim == 4 and latent2.ndim == 4:
b1, c1, f1, s1 = latent1.shape
b2, c2, f2, s2 = latent2.shape
if not (b1 == b2 and c1 == c2 and s1 == s2):
raise ValueError(
f"Latent dimensions must match (except frames dimension).\n"
f"Got shapes {latent1.shape} and {latent2.shape}"
)
else:
raise ValueError(
f"Latent dimensions must match (except frames dimension).\n"
f"Latent dimensions must be 4 (audio) or 5 (video) for both inputs.\n"
f"Got shapes {latent1.shape} and {latent2.shape}"
)
@@ -175,10 +196,10 @@ class LTXVAddLatents:
"""Merges noise masks from both latents with proper handling."""
if "noise_mask" in latents1 and "noise_mask" in latents2:
return torch.cat([latents1["noise_mask"], latents2["noise_mask"]], dim=2)
elif "noise_mask" in latents1:
elif "noise_mask" in latents1 and latents1["noise_mask"] is not None:
zeros = torch.zeros_like(latents1["noise_mask"][:, :, :frames2, :, :])
return torch.cat([latents1["noise_mask"], zeros], dim=2)
elif "noise_mask" in latents2:
elif "noise_mask" in latents2 and latents2["noise_mask"] is not None:
zeros = torch.zeros_like(latents2["noise_mask"][:, :, :frames1, :, :])
return torch.cat([zeros, latents2["noise_mask"]], dim=2)
return None
@@ -242,13 +263,22 @@ class LTXVSetVideoLatentNoiseMasks:
video_latent = s["samples"]
batch_size, channels, num_frames, height, width = video_latent.shape
# Initialize noise_mask if not present
# Initialize noise_mask if not present or resize if dimensions don't match
if "noise_mask" not in s:
s["noise_mask"] = torch.zeros(
(batch_size, 1, num_frames, height, width),
dtype=video_latent.dtype,
device=video_latent.device,
)
else:
existing_shape = s["noise_mask"].shape
# Check if noise_mask has the right number of frames
if existing_shape[2] != num_frames:
s["noise_mask"] = torch.zeros(
(batch_size, 1, num_frames, height, width),
dtype=video_latent.dtype,
device=video_latent.device,
)
# Process masks
masks_reshaped = self._reshape_masks(masks)
@@ -301,8 +331,74 @@ class LTXVSetVideoLatentNoiseMasks:
noise_mask[:, :, f] = resized_masks[mask_idx]
@comfy_node(name="LTXVDilateLatent")
class LTXVDilateLatent:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"latent": ("LATENT",),
"horizontal_scale": (
"INT",
{"default": 1, "min": 1, "max": 100, "step": 1},
),
"vertical_scale": (
"INT",
{"default": 1, "min": 1, "max": 100, "step": 1},
),
}
}
RETURN_TYPES = ("LATENT",)
FUNCTION = "dilate_latent"
CATEGORY = "latent/video"
DESCRIPTION = "Dilates a latent by a grid size."
def dilate_latent(
self, latent: dict, horizontal_scale: int, vertical_scale: int
) -> tuple:
if horizontal_scale == 1 and vertical_scale == 1:
return (latent,)
samples = latent["samples"]
mask = latent.get("noise_mask", None)
dilated_shape = samples.shape[:3] + (
samples.shape[3] * vertical_scale,
samples.shape[4] * horizontal_scale,
)
dilated_samples = torch.zeros(
dilated_shape,
device=samples.device,
dtype=samples.dtype,
requires_grad=False,
)
dilated_samples[..., ::vertical_scale, ::horizontal_scale] = samples
dilated_mask_shape = (
dilated_samples.shape[0],
1,
dilated_samples.shape[2],
dilated_samples.shape[3],
dilated_samples.shape[4],
)
dilated_mask = torch.full(
dilated_mask_shape,
-1.0,
device=samples.device,
dtype=samples.dtype,
requires_grad=False,
)
dilated_mask[..., ::vertical_scale, ::horizontal_scale] = (
mask if mask is not None else 1.0
)
latent = {"samples": dilated_samples, "noise_mask": dilated_mask}
return (latent,)
@comfy_node(name="LTXVAddLatentGuide")
class LTXVAddLatentGuide(nodes_lt.LTXVAddGuide):
class LTXVAddLatentGuide:
@classmethod
def INPUT_TYPES(s):
return {
@@ -340,7 +436,22 @@ class LTXVAddLatentGuide(nodes_lt.LTXVAddGuide):
):
noise_mask = nodes_lt.get_noise_mask(latent)
latent = latent["samples"]
guiding_latent = guiding_latent["samples"]
guide = guiding_latent["samples"]
assert (
latent.shape[4] % guide.shape[4] == 0
and latent.shape[3] % guide.shape[3] == 0
), "The ratio of the height and width of the latents and optional_guiding_latents must be an integer"
guiding_latent = LTXVDilateLatent().dilate_latent(
guiding_latent,
horizontal_scale=latent.shape[4] // guide.shape[4],
vertical_scale=latent.shape[3] // guide.shape[3],
)[0]
guide = guiding_latent["samples"]
guide_mask = guiding_latent.get("noise_mask", None)
scale_factors = vae.downscale_index_formula
if latent_idx <= 0:
@@ -348,15 +459,16 @@ class LTXVAddLatentGuide(nodes_lt.LTXVAddGuide):
else:
frame_idx = 1 + (latent_idx - 1) * scale_factors[0]
positive, negative, latent, noise_mask = self.append_keyframe(
positive, negative, latent, noise_mask = nodes_lt.LTXVAddGuide.append_keyframe(
positive=positive,
negative=negative,
frame_idx=frame_idx,
latent_image=latent,
noise_mask=noise_mask,
guiding_latent=guiding_latent,
guiding_latent=guide,
strength=strength,
scale_factors=scale_factors,
guide_mask=guide_mask,
)
return (
@@ -364,3 +476,355 @@ class LTXVAddLatentGuide(nodes_lt.LTXVAddGuide):
negative,
{"samples": latent, "noise_mask": noise_mask},
)
@comfy_node(name="LTXVImgToVideoConditionOnly")
class LTXVImgToVideoConditionOnly:
"""
Applies image conditioning to the first frames of a video latent.
Features:
- Takes existing latent and applies image conditioning
- Automatically resizes image to match latent dimensions
- Creates noise mask for strength control
- Supports bypass mode
"""
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"vae": ("VAE",),
"image": ("IMAGE",),
"latent": ("LATENT",),
"strength": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 1.0}),
},
"optional": {
"bypass": (
"BOOLEAN",
{"default": False, "tooltip": "Bypass the conditioning."},
),
},
}
RETURN_TYPES = ("LATENT",)
RETURN_NAMES = ("latent",)
CATEGORY = "conditioning/video_models"
FUNCTION = "generate"
DESCRIPTION = (
"Applies image conditioning to the first frames of an existing latent. "
"Creates a noise mask to control conditioning strength."
)
def encode_image(self, image, shape, vae):
time_scale_factor, height_scale_factor, width_scale_factor = (
vae.downscale_index_formula
)
batch, channels, frames, height, width = shape
width *= width_scale_factor
height *= height_scale_factor
# Resize image to match latent dimensions
if image.shape[1] != height or image.shape[2] != width:
pixels = comfy.utils.common_upscale(
image.movedim(-1, 1), width, height, "bilinear", "center"
).movedim(1, -1)
else:
pixels = image
# Encode image (only RGB channels)
encode_pixels = pixels[:, :, :, :3]
t = vae.encode(encode_pixels)
return t
def generate(self, image, vae, latent, strength, bypass=False):
if bypass:
return (latent,)
samples = latent["samples"]
t = self.encode_image(image, samples.shape, vae)
# Apply encoded image to first frames
samples[:, :, : t.shape[2]] = t
# Create noise mask for conditioning
conditioning_latent_frames_mask = torch.ones(
(1, 1, samples.shape[2], 1, 1),
dtype=torch.float32,
device=samples.device,
)
conditioning_latent_frames_mask[:, :, : t.shape[2]] = 1.0 - strength
return ({"samples": samples, "noise_mask": conditioning_latent_frames_mask},)
def get_video_latent_blend_coefficients(
video_frame_index_start,
video_frame_index_end,
video_frame_count,
slope_len=3,
):
"""
Returns a blend coefficient list shaped such that:
- It is 0.0 outside the range [video_frame_index_start, video_frame_index_end]
- It ramps up from 0.0 to 1.0 starting at video_frame_index_start-slope_len to video_frame_index_start
- It stays at 1.0 during [video_frame_index_start, video_frame_index_end]
- It ramps down from 1.0 to 0.0 starting at video_frame_index_end to video_frame_index_end+slope_len
- The 'slope_len' parameter controls the sharpness (in frames) of the slope.
"""
coeffs = [0.0] * video_frame_count
# Clamp arguments to safe range
video_frame_index_start = max(
0, min(video_frame_count - 1, video_frame_index_start)
)
video_frame_index_end = max(
video_frame_index_start, min(video_frame_count - 1, video_frame_index_end)
)
slope_len = max(1, slope_len)
# Ramp up
rampl_start = max(0, video_frame_index_start - slope_len)
for i in range(rampl_start, video_frame_index_start):
# Linear ramp, but could be smoother (sigmoid/cosine)
coeffs[i] = (i - rampl_start + 1) / slope_len
# Plateau
for i in range(video_frame_index_start, video_frame_index_end + 1):
coeffs[i] = 1.0
# Ramp down
rampr_end = min(video_frame_count, video_frame_index_end + slope_len + 1)
for i in range(video_frame_index_end + 1, rampr_end):
coeffs[i] = 1.0 - ((i - (video_frame_index_end + 1) + 1) / slope_len)
coeffs[i] = max(0.0, coeffs[i])
import numpy as np
num_coeffs = len(coeffs)
pixel_frame_length = (num_coeffs - 1) * 8 + 1
xp = np.array([0] + list(range(1, pixel_frame_length, 8)))
fp = np.array(coeffs)
# Calculate the upsampled coefficients using np.interp
pixel_frame_positions = np.arange(pixel_frame_length)
pixel_frame_coefficients = np.interp(pixel_frame_positions, xp, fp).tolist()
return coeffs, pixel_frame_coefficients
@comfy_node(description="LTXV Set Audio Video Mask By Time")
class LTXVSetAudioVideoMaskByTime:
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"av_latent": ("LATENT",),
"positive": ("CONDITIONING",),
"negative": ("CONDITIONING",),
"model": ("MODEL",),
"vae": ("VAE",),
"audio_vae": ("VAE",),
"start_time": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 2000.0}),
"end_time": ("FLOAT", {"default": 10.0, "min": 0.0, "max": 2000.0}),
"video_fps": ("FLOAT", {"default": 24.0, "min": 0.0, "max": 500.0}),
"mask_video": ("BOOLEAN", {"default": True}),
"mask_audio": ("BOOLEAN", {"default": True}),
"mask_init_value_video": (
"FLOAT",
{"default": 0.0, "min": 0.0, "max": 1.0},
),
"mask_init_value_audio": (
"FLOAT",
{"default": 0.0, "min": 0.0, "max": 1.0},
),
"slope_len": (
"INT",
{"default": 3, "min": 1, "max": 100, "step": 1},
),
},
"optional": {
"spatial_mask": (
"MASK",
{"default": None, "tooltip": "Spatial mask."},
),
},
}
RETURN_TYPES = ("CONDITIONING", "CONDITIONING", "LATENT", "FLOAT", "FLOAT")
RETURN_NAMES = (
"positive",
"negative",
"av_latent",
"video_latent_blend_coefficients",
"video_pixel_blend_coefficients",
)
FUNCTION = "run"
CATEGORY = "utility"
DESCRIPTION = "Sets the audio and video mask by time."
def run(
self,
av_latent,
positive,
negative,
model,
vae,
audio_vae,
start_time,
end_time,
video_fps,
mask_video,
mask_audio,
mask_init_value_video,
mask_init_value_audio,
slope_len,
spatial_mask=None,
):
from comfy.ldm.lightricks.av_model import LTXAVModel
if model.model.diffusion_model.__class__.__name__ != "LTXAVModel":
raise ValueError()
ltxav: LTXAVModel = model.model.diffusion_model
# Extract configuration from the audio VAE
sampling_rate = audio_vae.autoencoder.sampling_rate
mel_hop_length = audio_vae.autoencoder.mel_hop_length
audio_latents_per_second = (
sampling_rate / mel_hop_length / LATENT_DOWNSAMPLE_FACTOR
)
time_scale_factor = vae.downscale_index_formula[0]
video_latents_per_second = video_fps / time_scale_factor
if not isinstance(av_latent["samples"], NestedTensor):
raise ValueError("av_latent must be a NestedTensor")
video_samples, audio_samples = ltxav.separate_audio_and_video_latents(
av_latent["samples"].tensors,
None,
)
video_mask = torch.full(
(
video_samples.shape[0],
video_samples.shape[1],
video_samples.shape[2],
video_samples.shape[3],
video_samples.shape[4],
),
fill_value=mask_init_value_video,
)
audio_mask = torch.full(
(
audio_samples.shape[0],
audio_samples.shape[1],
audio_samples.shape[2],
audio_samples.shape[3],
),
fill_value=mask_init_value_audio,
)
if spatial_mask is not None:
if spatial_mask.ndim == 3:
spatial_mask = spatial_mask.unsqueeze(0)
if spatial_mask.ndim == 2:
spatial_mask = spatial_mask.unsqueeze(0).unsqueeze(0)
spatial_mask = torch.nn.functional.interpolate(
spatial_mask,
size=(video_samples.shape[3], video_samples.shape[4]),
mode="bilinear",
align_corners=False,
)
video_latent_frame_count = video_samples.shape[2]
audio_latent_frame_count = audio_samples.shape[2]
video_pixel_frame_count = (video_latent_frame_count - 1) * time_scale_factor + 1
xp = np.array(
[0]
+ list(
range(1, video_pixel_frame_count + time_scale_factor, time_scale_factor)
)
)
video_pixel_frame_start_raw = int(round(start_time * video_fps))
# video_frame_index_start = index of the value in xp rounding up
video_latent_frame_index_start = np.searchsorted(
xp, video_pixel_frame_start_raw, side="left"
)
video_pixel_frame_end_raw = int(round(end_time * video_fps))
# video_frame_index_end = index of the value in xp rounding down
video_latent_frame_index_end = (
np.searchsorted(xp, video_pixel_frame_end_raw, side="right") - 1
)
audio_latent_frame_index_start = int(
round(start_time * audio_latents_per_second)
)
audio_latent_frame_index_end = (
int(round(end_time * audio_latents_per_second)) + 1
)
# clamping
video_latent_frame_index_start = max(0, video_latent_frame_index_start)
video_latent_frame_index_end = min(
video_latent_frame_index_end, video_latent_frame_count
)
audio_latent_frame_index_start = max(0, audio_latent_frame_index_start)
audio_latent_frame_index_end = min(
audio_latent_frame_index_end, audio_latent_frame_count
)
print(
"noise mask start and end indices: video (%d %d), audio (%d %d), video fps: %f, video_latents_per_second: %f, audio_latents_per_second: %f, "
"video_latent_frame_count: %d, video_pixel_frame_count: %d, video_pixel_frame_start_raw: %d, video_pixel_frame_end_raw: %d, start_time: %f, end_time: %f"
% (
video_latent_frame_index_start,
video_latent_frame_index_end,
audio_latent_frame_index_start,
audio_latent_frame_index_end,
video_fps,
video_latents_per_second,
audio_latents_per_second,
video_latent_frame_count,
video_pixel_frame_count,
video_pixel_frame_start_raw,
video_pixel_frame_end_raw,
start_time,
end_time,
)
)
if mask_video:
if spatial_mask is not None:
video_mask[
:,
:,
video_latent_frame_index_start:video_latent_frame_index_end,
:,
:,
] = spatial_mask
else:
video_mask[
:, :, video_latent_frame_index_start:video_latent_frame_index_end
] = 1.0
if mask_audio:
audio_mask[
:, :, audio_latent_frame_index_start:audio_latent_frame_index_end
] = 1.0
av_latent["noise_mask"] = NestedTensor(
ltxav.recombine_audio_and_video_latents(video_mask, audio_mask)
)
video_latent_blend_coefficients, video_pixel_blend_coefficients = (
get_video_latent_blend_coefficients(
video_latent_frame_index_start,
video_latent_frame_index_end,
video_latent_frame_count,
slope_len=slope_len,
)
)
return (
positive,
negative,
av_latent,
video_latent_blend_coefficients,
video_pixel_blend_coefficients,
)
looping_sampler.md
+293
View File
@@ -0,0 +1,293 @@
# LTXVLoopingSampler Documentation
## Overview
The **LTXVLoopingSampler** is a unified ComfyUI node designed for generating long, high-resolution videos through versatile guidance and conditioningi techniques, as well as temporal and spatial tiling. It overcomes memory and computational limitations by breaking video generation into manageable chunks, enabling the creation of videos that would otherwise be impossible to generate in a single model run.
It provides a multitude of features (autoregressive long vidoes, keyframes, guidance using IC-LoRA video modalities, normalization, 'long memory' with conditioning on negative positional encodings, and more ) all in one place.
The **MultiPromptProvider** utility node for providing dynamic prompts per temporal tile.
---
## LTXVLoopingSampler
### Core Concept
The LTXVLoopingSampler addresses two fundamental challenges in video generation:
1. **Temporal Length**: Generating very long videos by dividing them into overlapping temporal segments
2. **Spatial Resolution**: Creating high-resolution frames by splitting them into overlapping spatial regions
Each "tile" is processed independently and then seamlessly blended with its neighbors to create the final coherent video output.
### Key Features
- **🎬 Temporal Tiling**: Generates long videos by processing overlapping time segments
- **🖼️ Spatial Tiling**: Creates high-resolution output by dividing frames into spatial regions
- **🎯 Multiple Conditioning Modes**: Supports image conditioning, guiding latents, and negative index conditioning
- **⚡ Memory Efficient**: Processes one spatial tile at a time to minimize memory usage
- **🔄 Seamless Blending**: Uses weighted blending for smooth transitions between tiles
- **🎲 Advanced Seeding**: Configurable per-tile seeding for reproducible results
### Interface Reference
#### Required Parameters
| Parameter | Type | Default | Range | Description |
|-----------|------|---------|-------|-------------|
| **model** | MODEL | - | - | The diffusion model for video generation |
| **vae** | VAE | - | - | VAE for encoding/decoding between pixel and latent space |
| **noise** | NOISE | - | - | Noise generator for the sampling process |
| **sampler** | SAMPLER | - | - | Sampling algorithm (e.g., Euler, DPM++) |
| **sigmas** | SIGMAS | - | - | Noise schedule defining the denoising steps |
| **guider** | GUIDER | - | - | Conditioning guider (must be STGGuiderAdvanced to work with LTXVLoopingSampler) |
| **latents** | LATENT | - | - | Input latent tensor defining video dimensions and length. Non empty latents can be passed here to enable a partial denoising use case. |
#### Temporal Configuration
| Parameter | Type | Default | Range | Description |
|-----------|------|---------|-------|-------------|
| **temporal_tile_size** | INT | 80 | 24-1000 (step: 8) | Frames per temporal tile (in pixel space). The first tile will be one frame longer than this number, and all new tiles will add `temporal_tile_size-temporal_overlap` new frames each. |
| **temporal_overlap** | INT | 24 | 16-80 (step: 8) | Overlapping frames between consecutive tiles |
| **temporal_overlap_cond_strength** | FLOAT | 0.5 | 0.0-1.0 | How strongly the final frames of the previous tile condition the new tile generation |
#### Spatial Configuration
| Parameter | Type | Default | Range | Description |
|-----------|------|---------|-------|-------------|
| **horizontal_tiles** | INT | 1 | 1-6 | Number of horizontal spatial divisions |
| **vertical_tiles** | INT | 1 | 1-6 | Number of vertical spatial divisions |
| **spatial_overlap** | INT | 1 | 1-8 | Overlapping latent 'pixels' between spatial tiles (latent space) |
#### Conditioning Controls
| Parameter | Type | Default | Range | Description |
|-----------|------|---------|-------|-------------|
| **guiding_strength** | FLOAT | 1.0 | 0.0-1.0 | Influence of guiding latents on generation, if provided. |
| **cond_image_strength** | FLOAT | 1.0 | 0.0-1.0 | Influence of conditioning images on first image in classical i2v setup, and of all keyframes. |
#### Optional Advanced Parameters
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| **optional_cond_images** | IMAGE | None | Images for I2V conditioning (resized to match output using center crop operation) |
| **optional_guiding_latents** | LATENT | None | Latents for guided generation (usually used with IC-LoRA loaded) |
| **adain_factor** | FLOAT | 0.0 (0.0-1.0) | AdaIn normalization strength to prevent oversaturation. AdaIn is applied on each new temporal tile, with respect to the latent statistics of the first temporal tile. |
| **optional_positive_conditionings** | CONDITIONING | None | Per-tile prompts (from MultiPromptProvider) |
| **optional_negative_index_latents** | LATENT | None | Context latents for long-term coherence |
| **guiding_start_step** | INT | 0 (0-1000) | Step to begin applying guiding latents, if provided. |
| **guiding_end_step** | INT | 1000 (0-1000) | Step to stop applying guiding latents, if provided. |
| **optional_cond_image_indices** | STRING | "0" | Keyframe indices for conditioning (comma-separated list of indices) |
| **optional_normalizing_latents** | LATENT | None | Reference latents for output normalization. If provided, used in the AdaIn operation, each spatio-temporal tile with its counterpart, instead of using the first temporal tile as a normalization refernece. |
### Output
| Output | Type | Description |
|--------|------|-------------|
| **denoised_output** | LATENT | Complete video latents after tiling and blending |
### Generation Modes
#### 🎨 Text-to-Video (T2V)
Pure text-driven video generation starting from noise.
**Setup:**
- Don't connect `optional_cond_images`
- Provide text conditioning through the guider
- Video generates from pure noise
#### 🖼️ Image-to-Video (I2V)
Video generation conditioned on input images.
**Setup:**
- Connect a single image to `optional_cond_images` and set `optional_cond_image_indices` to "0".
- Images are automatically resized to match output dimensions
#### 🗝️ Keyframe-Based Generation
Keyframe-based generation allows you to guide the video at specific frames using conditioning images (keyframes). This is useful for tasks like animating a sequence between important visual states or ensuring the video matches certain reference images at chosen points.
- Use `optional_cond_images` to provide one or more keyframe images.
- Use `optional_cond_image_indices` to specify which frames should be conditioned on these images. This should be a comma-separated list of frame indices (e.g., `"0, 32, 64"`).
- Each image in `optional_cond_images` will be applied to the corresponding frame index in `optional_cond_image_indices`.
- The node will automatically resize and center-crop the images to match the output dimensions.
**Example:**
- To condition the first and last frames of a 64-frame video, set:
- `optional_cond_images`: [start_image, end_image]
- `optional_cond_image_indices`: `"0, 63"`
This will ensure the generated video starts and ends with the provided keyframes, smoothly interpolating between them.
**Tip:** You can combine keyframe conditioning with text prompts and guiding latents for even more control over the video generation process.
#### 🎯 Guided Generation
Advanced control using guiding latents (often with IC-LoRA).
**Setup:**
- Connect guiding latents to `optional_guiding_latents`
- Adjust `guiding_strength` (0.0-1.0)
- Set `guiding_start_step` and `guiding_end_step` for control over the effect of the guidance during the denoising steps.
#### 📏 High-Resolution Generation
Create videos larger than base model resolution using spatial tiling.
**Configuration Example:**
```
horizontal_tiles: 2
vertical_tiles: 2
spatial_overlap: 2
```
This creates a 2x2 grid of spatial tiles with 2-pixel overlap for seamless blending.
#### ⏱️ Long Video Generation
Generate extended sequences using temporal tiling.
**How It Works:**
- Each temporal tile is conditioned on the **end frames** of the previous tile
- This creates seamless motion continuity across tile boundaries
- The `temporal_overlap` defines how many frames from the previous tile are used for conditioning
**Best Practices:**
- Use `temporal_overlap` = 1/3 of `temporal_tile_size`
- Higher `temporal_overlap_cond_strength` = stronger temporal consistency
- Set `adain_factor` = 0.1-0.3 for long sequences to prevent oversaturation
- Consider using different prompts per tile via MultiPromptProvider
#### 🧩 Negative Index Latents for Enhanced Coherence
You can provide `optional_negative_index_latents` to further improve long video consistency. This feature allows each temporal chunk to attend to an additional latent (or set of latents), typically derived from another image or video, to guide the generation process and maintain global coherence.
- **How it works:**
Each temporal chunk is conditioned on the provided negative index latent(s) using *negative positional embeddings*. This mechanism helps the model reference global context or style cues throughout the video, reducing drift and improving overall consistency, especially in long sequences.
- **Usage:**
- Connect your reference latent(s) to the `optional_negative_index_latents` input.
- **Typical applications:**
- Maintaining a consistent subject or style across long videos
- Referencing a global template or anchor image
- Enforcing scene or character coherence
**Tip:** You can combine negative index latents with keyframes, guiding latents, and text prompts for maximum control.
### Technical Implementation
#### Temporal Tiling Algorithm
The node processes video in overlapping temporal chunks, with each tile conditioned on the preceding tile's output:
1. **First Tile (0 → temporal_tile_size)**
- Uses `LTXVInContextSampler` (with guidance) or `LTXVBaseSampler` (without)
- Establishes the foundation for the video sequence
- Generated from pure noise (T2V) or conditioning images (I2V / keyframes)
2. **Subsequent Tiles**
- Uses `LTXVExtendSampler` to extend from previous tile's results
- **Conditioning Mechanism**: Each new tile is conditioned on the **final frames** (overlap region) of the preceding tile
- The overlap region from the previous tile provides temporal continuity and motion consistency
- `temporal_overlap_cond_strength` controls how strongly the previous tile's end frames influence the new generation
- Each tile advances by `(temporal_tile_size - temporal_overlap)` frames, creating seamless temporal progression
- Guiding latents, if provided, are used outside of the overlap region.
- Conditioning images (`optional_cond_images`), if provided, are used to condition the generation as well.
#### Spatial Tiling Algorithm
For high-resolution generation, frames are divided spatially:
1. **Tile Calculation**: Frame divided into overlapping regions
2. **Independent Processing**: Each spatial region processed separately
3. **Weight Generation**: Linear blending weights created for overlap regions
4. **Final Compositing**: Weighted accumulation produces seamless result
#### Seed Management
Each tile receives a unique, deterministic seed:
```
tile_seed = base_seed + temporal_offset + spatial_offset + custom_offset (hidedn interface)
```
This ensures:
- Reproducible results across runs
- Consistent randomness per tile
- Ability to re-generate specific tiles
#### Memory Optimization Strategies
- **Sequential Spatial Processing**: Only one spatial tile in memory at a time
- **Temporal Chunking**: Long videos processed in manageable segments
- **Weighted Accumulation**: Final output built incrementally
- **Garbage Collection**: Intermediate results freed after use
### Configuration Guidelines
#### Temporal Parameters
- **Short Videos (< 5 seconds)**: `temporal_tile_size: 40-80`
- **Medium Videos (5-15 seconds)**: `temporal_tile_size: 80-120`
- **Long Videos (> 15 seconds)**: `temporal_tile_size: 120-200`
- **Overlap Rule**: Use 25-30% of tile size for smooth transitions
#### Spatial Parameters
- **Standard Resolution**: `horizontal_tiles: 1, vertical_tiles: 1`
- **2K Generation**: `horizontal_tiles: 2, vertical_tiles: 1-2`
- **4K Generation**: `horizontal_tiles: 2-3, vertical_tiles: 2-3`
- **Overlap Minimum**: At least 1 pixel, increase for better blending
#### Quality Settings
- **High Quality**: Lower `adain_factor` (0.0-0.1), higher overlap
- **Fast Generation**: Higher `adain_factor` (0.2-0.5), lower overlap
- **Temporal Consistency**: Higher `temporal_overlap_cond_strength`
### Common Use Cases
#### Creating Cinematic Sequences
```
temporal_tile_size: 80
temporal_overlap: 24
horizontal_tiles: 2
vertical_tiles: 1
adain_factor: 0.1
```
#### Long-Form Content Generation
```
temporal_tile_size: 120
temporal_overlap: 40
adain_factor: 0.2
+ MultiPromptProvider for narrative progression
```
#### High-Resolution Showcase Videos
```
horizontal_tiles: 3
vertical_tiles: 2
spatial_overlap: 3
temporal_tile_size: 60
```
### Troubleshooting
#### Common Issues
- **Visible Seams**: Increase spatial/temporal overlap
- **Oversaturation**: Increase `adain_factor`
- **Memory Errors**: Reduce tile sizes or use fewer spatial tiles
- **Inconsistent Motion**: Increase `temporal_overlap` and `temporal_overlap_cond_strength`
- **Temporal Discontinuities**: The end frames of each tile condition the next tile - ensure sufficient overlap
- **Color Shifts**: Use `optional_normalizing_latents` for reference and/or increase `adain_factor`
#### Performance Tips
- Process spatial tiles sequentially to minimize memory usage
- Use appropriate tile sizes for your hardware capabilities
- Consider temporal overlap vs. generation speed trade-offs
---
## MultiPromptProvider
A companion utility node that enables dynamic prompt changes across temporal tiles.
- Connect multiple pipe-separated prompts (`"prompt1|prompt2|prompt3"`) to create evolving narratives throughout video generation.
- Each prompt is used in one temporal_tile in `LTXVLoopingSampler`.
- If not enough prompts are provided, the last one is repeated.
- If too many prompts provided, the last ones are ignored.
---
looping_sampler.py
+341 -93
View File
@@ -5,7 +5,7 @@ import comfy
import torch
from .easy_samplers import LTXVBaseSampler, LTXVExtendSampler, LTXVInContextSampler
from .latents import LTXVSelectLatents
from .latents import LTXVDilateLatent, LTXVSelectLatents
from .nodes_registry import comfy_node
@@ -16,7 +16,9 @@ class TileConfig:
tile_latents: dict
tile_guiding_latents: dict
tile_negative_index_latents: dict
tile_cond_image: torch.Tensor
tile_keyframes: torch.Tensor
keyframe_per_tile_indices: list
tile_normalizing_latents: dict
tile_height: int
tile_width: int
v: int
@@ -33,6 +35,7 @@ class SamplingConfig:
temporal_tile_size: int
temporal_overlap: int
temporal_overlap_cond_strength: float
cond_image_strength: float
guiding_strength: float
adain_factor: float
optional_negative_index: int
@@ -42,7 +45,8 @@ class SamplingConfig:
width_scale_factor: int
height_scale_factor: int
per_tile_seed_offsets: list
per_tile_use_negative_latents: list
guiding_start_step: int
guiding_end_step: int
@dataclass
@@ -81,16 +85,6 @@ class LTXVLoopingSampler:
"tooltip": "The latents to use for creating the long video, they can be guiding latents or empty latents when no guidance is used."
},
),
"guiding_strength": (
"FLOAT",
{
"default": 1.0,
"min": 0.0,
"max": 1.0,
"step": 0.01,
"tooltip": "The strength of the conditioning on guiding latents, when optional_guiding_latents are provided.",
},
),
"temporal_tile_size": (
"INT",
{
@@ -111,6 +105,16 @@ class LTXVLoopingSampler:
"tooltip": "The overlap between the temporal tiles, in pixel frames.",
},
),
"guiding_strength": (
"FLOAT",
{
"default": 1.0,
"min": 0.0,
"max": 1.0,
"step": 0.01,
"tooltip": "The strength of the conditioning on guiding latents, when optional_guiding_latents are provided.",
},
),
"temporal_overlap_cond_strength": (
"FLOAT",
{
@@ -121,6 +125,16 @@ class LTXVLoopingSampler:
"tooltip": "The strength of the conditioning on the latents from the previous temporal tile.",
},
),
"cond_image_strength": (
"FLOAT",
{
"default": 1.0,
"min": 0.0,
"max": 1.0,
"step": 0.01,
"tooltip": "The strength of the conditioning on the optional_cond_images.",
},
),
"horizontal_tiles": (
"INT",
{
@@ -150,7 +164,7 @@ class LTXVLoopingSampler:
),
},
"optional": {
"optional_cond_image": (
"optional_cond_images": (
"IMAGE",
{
"tooltip": "The image to use for conditioning the first frame in the video (i2v setup). If not provided, the first frame will be unconditioned (t2v setup). The image will be resized to the size of the first frame."
@@ -184,6 +198,37 @@ class LTXVLoopingSampler:
"tooltip": "Special optional latents to condition on a negative index before each new temporal tile as a way to provide long term context during video generation."
},
),
"guiding_start_step": (
"INT",
{
"default": 0,
"min": 0,
"max": 1000,
"tooltip": "The step at which the guiding latents start to be used.",
},
),
"guiding_end_step": (
"INT",
{
"default": 1000,
"min": 0,
"max": 1000,
"tooltip": "The step at which the guiding latents stop to be used.",
},
),
"optional_cond_image_indices": (
"STRING",
{
"default": "0",
"tooltip": "The indices of the keyframes to use for the sampling, separated by commas. The indices are counted from the end of the video.",
},
),
"optional_normalizing_latents": (
"LATENT",
{
"tooltip": "The latents to use for normalizing the output latents, they will be used to normalize the output latents to the same statistics as the input latents."
},
),
},
}
@@ -198,14 +243,21 @@ class LTXVLoopingSampler:
if latent_dict is None:
return None
tile_samples = latent_dict["samples"][:, :, :, v_start:v_end, h_start:h_end]
return {"samples": tile_samples}
if "noise_mask" in latent_dict and latent_dict["noise_mask"] is not None:
tile_masks = latent_dict["noise_mask"][
:, :, :, v_start:v_end, h_start:h_end
]
return {"samples": tile_samples, "noise_mask": tile_masks}
else:
return {"samples": tile_samples}
def _extract_spatial_tile(
self,
samples,
latents,
optional_guiding_latents,
optional_negative_index_latents,
optional_cond_image,
optional_normalizing_latents,
optional_keyframes,
v_start,
v_end,
h_start,
@@ -216,7 +268,7 @@ class LTXVLoopingSampler:
"""Extract spatial tiles from all inputs for a given spatial region."""
# Extract spatial tile from latents
tile_latents = self._extract_latent_spatial_tile(
{"samples": samples}, v_start, v_end, h_start, h_end
latents, v_start, v_end, h_start, h_end
)
# Extract spatial tile from guiding latents if provided
@@ -229,25 +281,30 @@ class LTXVLoopingSampler:
optional_negative_index_latents, v_start, v_end, h_start, h_end
)
# Extract spatial tile from conditioning image if provided
tile_cond_image = None
if optional_cond_image is not None:
# Extract spatial tile from normalizing latents if provided
tile_normalizing_latents = self._extract_latent_spatial_tile(
optional_normalizing_latents, v_start, v_end, h_start, h_end
)
if optional_keyframes is not None:
# Scale coordinates for image
img_h_start = v_start * height_scale_factor
img_h_end = v_end * height_scale_factor
img_w_start = h_start * width_scale_factor
img_w_end = h_end * width_scale_factor
# Extract image tile
tile_cond_image = optional_cond_image[
tile_keyframes = optional_keyframes[
:, img_h_start:img_h_end, img_w_start:img_w_end, :
]
else:
tile_keyframes = None
return (
tile_latents,
tile_guiding_latents,
tile_negative_index_latents,
tile_cond_image,
tile_keyframes,
tile_normalizing_latents,
)
def _process_temporal_chunks(
@@ -309,9 +366,28 @@ class LTXVLoopingSampler:
seed_offset = self._get_per_tile_value(
sampling_config.per_tile_seed_offsets, i_temporal_tile
)
use_negative_latents = self._get_per_tile_value(
sampling_config.per_tile_use_negative_latents, i_temporal_tile
)
if tile_config.tile_normalizing_latents is not None:
normalizing_latent_chunk = LTXVSelectLatents().select_latents(
tile_config.tile_normalizing_latents,
start_index,
min(
end_index - 1,
tile_config.tile_normalizing_latents["samples"].shape[2] - 1,
),
)[0]
normalize_per_frame = True
print(
"Normalizing latents provided, normalizing per frame and channel with factor",
sampling_config.adain_factor,
)
else:
normalizing_latent_chunk = first_tile_out_latents
normalize_per_frame = False
print(
"No normalizing latents provided, normalizing per channel using first chunk with factor",
sampling_config.adain_factor,
)
model_config.noise.seed = self._calculate_tile_seed(
tile_config.first_seed,
@@ -330,6 +406,30 @@ class LTXVLoopingSampler:
chunk_index,
)
this_chunk_keyframe_indices = [
in_tile_index
for (tile_index, in_tile_index) in tile_config.keyframe_per_tile_indices
if tile_index == i_temporal_tile
]
if this_chunk_keyframe_indices and tile_config.tile_keyframes is not None:
this_chunk_keyframes = torch.cat(
[
tile_config.tile_keyframes[i_keyframe].unsqueeze(0)
for (i_keyframe, (tile_index, in_tile_index)) in enumerate(
tile_config.keyframe_per_tile_indices
)
if tile_index == i_temporal_tile
]
)
print(
f"Chunk {i_temporal_tile} keyframe indices: {this_chunk_keyframe_indices}"
)
else:
this_chunk_keyframes = None
print(f"Chunk {i_temporal_tile} has no keyframes")
this_chunk_keyframe_indices = ",".join(
[str(i) for i in this_chunk_keyframe_indices]
)
if start_index == 0:
if tile_config.tile_guiding_latents is not None:
tile_out_latents = LTXVInContextSampler().sample(
@@ -339,18 +439,17 @@ class LTXVLoopingSampler:
sigmas=model_config.sigmas,
noise=model_config.noise,
guiding_latents=guiding_latent_chunk,
optional_cond_image=tile_config.tile_cond_image,
optional_cond_images=this_chunk_keyframes,
optional_cond_indices=this_chunk_keyframe_indices,
num_frames=-1,
optional_negative_index_latents=(
tile_config.tile_negative_index_latents
if use_negative_latents
else None
),
optional_negative_index_latents=tile_config.tile_negative_index_latents,
optional_negative_index=sampling_config.optional_negative_index,
optional_negative_index_strength=sampling_config.optional_negative_index_strength,
optional_initialization_latents=latent_chunk,
optional_cond_strength=sampling_config.temporal_overlap_cond_strength,
optional_guiding_strength=sampling_config.guiding_strength,
cond_image_strength=sampling_config.cond_image_strength,
guiding_strength=sampling_config.guiding_strength,
guiding_start_step=sampling_config.guiding_start_step,
guiding_end_step=sampling_config.guiding_end_step,
)[0]
else:
tile_out_latents = LTXVBaseSampler().sample(
@@ -373,19 +472,17 @@ class LTXVLoopingSampler:
* sampling_config.width_scale_factor,
height=tile_config.tile_height
* sampling_config.height_scale_factor,
optional_cond_images=tile_config.tile_cond_image,
optional_cond_indices="0",
optional_cond_images=this_chunk_keyframes,
optional_cond_indices=this_chunk_keyframe_indices,
crop="center",
crf=30,
strength=sampling_config.temporal_overlap_cond_strength,
optional_negative_index_latents=(
tile_config.tile_negative_index_latents
if use_negative_latents
else None
),
strength=sampling_config.cond_image_strength,
optional_negative_index_latents=tile_config.tile_negative_index_latents,
optional_negative_index=sampling_config.optional_negative_index,
optional_negative_index_strength=sampling_config.optional_negative_index_strength,
optional_initialization_latents=latent_chunk,
guiding_start_step=sampling_config.guiding_start_step,
guiding_end_step=sampling_config.guiding_end_step,
)[0]
first_tile_out_latents = copy.deepcopy(tile_out_latents)
else:
@@ -406,17 +503,19 @@ class LTXVLoopingSampler:
guider=new_guider,
strength=sampling_config.temporal_overlap_cond_strength,
guiding_strength=sampling_config.guiding_strength,
cond_image_strength=sampling_config.cond_image_strength,
optional_guiding_latents=guiding_latent_chunk,
optional_reference_latents=first_tile_out_latents,
optional_cond_images=this_chunk_keyframes,
optional_cond_indices=this_chunk_keyframe_indices,
optional_reference_latents=normalizing_latent_chunk,
normalize_per_frame=normalize_per_frame,
adain_factor=sampling_config.adain_factor,
optional_negative_index_latents=(
tile_config.tile_negative_index_latents
if use_negative_latents
else None
),
optional_negative_index_latents=tile_config.tile_negative_index_latents,
optional_negative_index=sampling_config.optional_negative_index,
optional_negative_index_strength=sampling_config.optional_negative_index_strength,
optional_initialization_latents=latent_chunk,
guiding_start_step=sampling_config.guiding_start_step,
guiding_end_step=sampling_config.guiding_end_step,
)[0]
chunk_index += 1
@@ -424,25 +523,33 @@ class LTXVLoopingSampler:
return tile_out_latents
def _create_spatial_weights(
self, tile_samples, v, h, horizontal_tiles, vertical_tiles, spatial_overlap
self,
tile_shape,
v,
h,
horizontal_tiles,
vertical_tiles,
spatial_overlap,
device,
dtype,
):
"""Create blending weights for spatial tiles."""
tile_weights = torch.ones_like(tile_samples, device=tile_samples.device)
tile_weights = torch.ones(tile_shape, device=device, dtype=dtype)
# Apply horizontal blending weights
if h > 0: # Left overlap
h_blend = torch.linspace(0, 1, spatial_overlap, device=tile_samples.device)
h_blend = torch.linspace(0, 1, spatial_overlap, device=device, dtype=dtype)
tile_weights[:, :, :, :, :spatial_overlap] *= h_blend.view(1, 1, 1, 1, -1)
if h < horizontal_tiles - 1: # Right overlap
h_blend = torch.linspace(1, 0, spatial_overlap, device=tile_samples.device)
h_blend = torch.linspace(1, 0, spatial_overlap, device=device, dtype=dtype)
tile_weights[:, :, :, :, -spatial_overlap:] *= h_blend.view(1, 1, 1, 1, -1)
# Apply vertical blending weights
if v > 0: # Top overlap
v_blend = torch.linspace(0, 1, spatial_overlap, device=tile_samples.device)
v_blend = torch.linspace(0, 1, spatial_overlap, device=device, dtype=dtype)
tile_weights[:, :, :, :spatial_overlap, :] *= v_blend.view(1, 1, 1, -1, 1)
if v < vertical_tiles - 1: # Bottom overlap
v_blend = torch.linspace(1, 0, spatial_overlap, device=tile_samples.device)
v_blend = torch.linspace(1, 0, spatial_overlap, device=device, dtype=dtype)
tile_weights[:, :, :, -spatial_overlap:, :] *= v_blend.view(1, 1, 1, -1, 1)
return tile_weights
@@ -470,11 +577,22 @@ class LTXVLoopingSampler:
"""Get a value from a per-tile configuration list, using the last value if the list is shorter."""
return value_list[min(tile_index, len(value_list) - 1)]
def _parse_per_tile_config(self, config_string, default_value, converter_func):
"""Parse a comma-separated per-tile configuration string into a list with type conversion."""
def _parse_comma_separated_string(
self, config_string, default_value, converter_func, optional_total_size=None
):
"""Parse a comma-separated configuration string into a list with type conversion."""
if config_string == "":
config_string = default_value
return [converter_func(item.strip()) for item in config_string.split(",")]
values = [converter_func(item.strip()) for item in config_string.split(",")]
def handle_negative_index(value):
if value < 0:
return value + optional_total_size
return value
if optional_total_size is not None:
values = [handle_negative_index(value) for value in values]
return values
def _prepare_guider_for_chunk(
self, guider, optional_positive_conditionings, chunk_index
@@ -499,6 +617,78 @@ class LTXVLoopingSampler:
else:
return guider
def _calculate_keyframe_per_tile_indices(
self, keyframe_indices, temporal_tile_size, temporal_overlap, num_frames
):
"""
Calculate which temporal tile each keyframe falls into.
Returns a list of tuples (temporal_tile_index, in_tile_index) for each keyframe.
- temporal_tile_index: which temporal tile the keyframe falls in
- in_tile_index: index within that specific temporal tile
First tile: frames [0, temporal_tile_size - 8] (size = temporal_tile_size - 8 + 1)
Subsequent tiles follow the pattern from _process_temporal_chunks:
- Tile n starts at: n * (temporal_tile_size - temporal_overlap)
- Tile n ends at: temporal_tile_size + n * (temporal_tile_size - temporal_overlap) - 1
For subsequent tiles, keyframes with in_tile_index < temporal_overlap
don't count as falling in that tile.
"""
result = []
for keyframe_index in keyframe_indices:
if keyframe_index >= num_frames:
print(
f"Keyframe index {keyframe_index} is greater than num_frames {num_frames}, skipping"
)
continue
# First tile (tile 0): covers frames [0, temporal_tile_size - 8]
if keyframe_index < temporal_tile_size - 7:
result.append((0, keyframe_index))
continue
# Find which subsequent tile this keyframe could fall into
# Tile n starts at: n * (temporal_tile_size - temporal_overlap) - 7
# Tile n ends at: temporal_tile_size + n * (temporal_tile_size - temporal_overlap) - 1 - 7
tile_step = temporal_tile_size - temporal_overlap
tile_index = 1
while True:
tile_start = tile_index * tile_step - 7
tile_end = temporal_tile_size + tile_index * tile_step - 1 - 7
print(
f"Tile {tile_index} starts at {tile_start} and ends at {tile_end}"
)
# Check if keyframe falls within this tile's range
if keyframe_index <= tile_end:
in_tile_index = (
keyframe_index - tile_start - 7
) # this extra -7 is needed because the first latent is re-interpreted inside the temporal tile as 1 pixel frame
# For tiles > 0, if in_tile_index < temporal_overlap,
# the keyframe doesn't "fall" in this tile - assign to previous tile
if in_tile_index < temporal_overlap:
tile_index -= 1
if tile_index == 0:
# Previous tile is the first tile
in_tile_index = keyframe_index
else:
# Previous tile starts at (tile_index) * tile_step
prev_start = tile_start - tile_step
in_tile_index = (
keyframe_index - prev_start - 7
) # same reason as above
result.append((tile_index, in_tile_index))
break
tile_index += 1
return result
def sample(
self,
model,
@@ -516,18 +706,27 @@ class LTXVLoopingSampler:
horizontal_tiles,
vertical_tiles,
spatial_overlap,
optional_cond_image=None,
optional_cond_images=None,
cond_image_strength=1.0,
optional_guiding_latents=None,
optional_negative_index_latents=None,
optional_negative_index=-1,
optional_negative_index_strength=1.0,
optional_negative_index_strength=1.0, # hidden interface
optional_positive_conditionings=None,
per_tile_seed_offsets="0",
per_tile_use_negative_latents="1",
guiding_start_step=0,
guiding_end_step=1000,
optional_cond_image_indices="0",
optional_normalizing_latents=None,
per_tile_seed_offsets="0", # hidden interface
):
# Get dimensions and prepare for spatial tiling
samples = latents["samples"]
if (
isinstance(samples, comfy.nested_tensor.NestedTensor)
and len(samples.tensors) == 2
):
raise ValueError(
"LoopingSampler currently does not support Audio Visual latents. please only use video latents."
)
batch, channels, frames, height, width = samples.shape
time_scale_factor, width_scale_factor, height_scale_factor = (
vae.downscale_index_formula
@@ -536,18 +735,52 @@ class LTXVLoopingSampler:
temporal_overlap = temporal_overlap // time_scale_factor
first_seed = noise.seed
per_tile_seed_offsets = self._parse_per_tile_config(
per_tile_seed_offsets = self._parse_comma_separated_string(
per_tile_seed_offsets, "0", int
)
per_tile_use_negative_latents = self._parse_per_tile_config(
per_tile_use_negative_latents, "1", lambda x: bool(int(x))
keyframe_indices = self._parse_comma_separated_string(
optional_cond_image_indices,
"0",
int,
optional_total_size=frames * time_scale_factor - 7,
)
keyframe_per_tile_indices = self._calculate_keyframe_per_tile_indices(
keyframe_indices,
temporal_tile_size * time_scale_factor,
temporal_overlap * time_scale_factor,
frames * time_scale_factor - 7,
)
print(f"Keyframe per tile indices: {keyframe_per_tile_indices}")
if optional_cond_images is not None:
optional_keyframes = (
comfy.utils.common_upscale(
optional_cond_images.movedim(-1, 1),
width * width_scale_factor,
height * height_scale_factor,
"bilinear",
crop="center",
)
.movedim(1, -1)
.clamp(0, 1)
)
else:
optional_keyframes = None
if optional_guiding_latents is not None:
guide = optional_guiding_latents["samples"]
assert (
latents["samples"].shape[2]
== optional_guiding_latents["samples"].shape[2]
samples.shape[2] == guide.shape[2]
), "The number of frames in the latents and optional_guiding_latents must be the same"
assert (
samples.shape[3] % guide.shape[3] == 0
and samples.shape[4] % guide.shape[4] == 0
), "The ratio of the height and width of the latents and optional_guiding_latents must be an integer"
grid_size_h = samples.shape[3] // guide.shape[3]
grid_size_w = samples.shape[4] // guide.shape[4]
optional_guiding_latents = LTXVDilateLatent().dilate_latent(
optional_guiding_latents, grid_size_w, grid_size_h
)[0]
# Calculate tile sizes with overlap
base_tile_height = (
@@ -557,21 +790,9 @@ class LTXVLoopingSampler:
width + (horizontal_tiles - 1) * spatial_overlap
) // horizontal_tiles
# Initialize output tensor and weight tensor
final_output = torch.zeros_like(samples, device=samples.device)
weights = torch.zeros_like(samples, device=samples.device)
if optional_cond_image is not None:
img_height = height * height_scale_factor
img_width = width * width_scale_factor
optional_cond_image = comfy.utils.common_upscale(
optional_cond_image.movedim(-1, 1),
img_width,
img_height,
"bicubic",
crop="center",
).movedim(1, -1)
img_batch, img_height, img_width, img_channels = optional_cond_image.shape
# Output tensors will be initialized after first tile to get correct temporal dimension
final_output = None
weights = None
# Process each spatial tile
for v in range(vertical_tiles):
@@ -605,12 +826,14 @@ class LTXVLoopingSampler:
tile_latents,
tile_guiding_latents,
tile_negative_index_latents,
tile_cond_image,
tile_keyframes,
tile_normalizing_latents,
) = self._extract_spatial_tile(
samples,
latents,
optional_guiding_latents,
optional_negative_index_latents,
optional_cond_image,
optional_normalizing_latents,
optional_keyframes,
v_start,
v_end,
h_start,
@@ -624,7 +847,9 @@ class LTXVLoopingSampler:
tile_latents=tile_latents,
tile_guiding_latents=tile_guiding_latents,
tile_negative_index_latents=tile_negative_index_latents,
tile_cond_image=tile_cond_image,
tile_keyframes=tile_keyframes,
keyframe_per_tile_indices=keyframe_per_tile_indices,
tile_normalizing_latents=tile_normalizing_latents,
tile_height=tile_height,
tile_width=tile_width,
v=v,
@@ -638,16 +863,22 @@ class LTXVLoopingSampler:
temporal_tile_size=temporal_tile_size,
temporal_overlap=temporal_overlap,
temporal_overlap_cond_strength=temporal_overlap_cond_strength,
cond_image_strength=cond_image_strength,
guiding_strength=guiding_strength,
adain_factor=adain_factor,
optional_negative_index=optional_negative_index,
optional_negative_index=(
-1 * tile_negative_index_latents["samples"].shape[2]
if tile_negative_index_latents is not None
else -1
),
optional_negative_index_strength=optional_negative_index_strength,
optional_positive_conditionings=optional_positive_conditionings,
time_scale_factor=time_scale_factor,
width_scale_factor=width_scale_factor,
height_scale_factor=height_scale_factor,
per_tile_seed_offsets=per_tile_seed_offsets,
per_tile_use_negative_latents=per_tile_use_negative_latents,
guiding_start_step=guiding_start_step,
guiding_end_step=guiding_end_step,
)
model_config = ModelConfig(
@@ -665,19 +896,36 @@ class LTXVLoopingSampler:
model_config,
)
# Create weight mask for this spatial tile
# Initialize output tensors on first tile (to get correct temporal dimension)
if final_output is None:
out_temporal = tile_out_latents["samples"].shape[2]
final_output = torch.zeros(
batch,
channels,
out_temporal,
height,
width,
device=samples.device,
dtype=samples.dtype,
)
weights = torch.zeros_like(final_output)
# Move tile samples to output device and create weight mask
tile_samples = tile_out_latents["samples"].to(final_output.device)
tile_weights = self._create_spatial_weights(
tile_latents["samples"],
tile_samples.shape,
v,
h,
horizontal_tiles,
vertical_tiles,
spatial_overlap,
final_output.device,
final_output.dtype,
)
# Add weighted tile to final output
final_output[:, :, :, v_start:v_end, h_start:h_end] += (
tile_out_latents["samples"].to(final_output.device) * tile_weights
tile_samples * tile_weights
)
weights[:, :, :, v_start:v_end, h_start:h_end] += tile_weights
low_vram_loaders.py
+127
View File
@@ -0,0 +1,127 @@
"""
Low VRAM loader nodes for sequential model loading.
These nodes add a 'dependencies' input to control execution order,
ensuring models load one after another instead of simultaneously.
This reduces peak VRAM usage in memory-constrained environments.
"""
import comfy.utils
import folder_paths
import nodes
from comfy.ldm.lightricks.vae.audio_vae import AudioVAE
from comfy_api.latest import io
from comfy_extras.nodes_hunyuan import LatentUpscaleModelLoader
class AnyType(str):
"""Custom type that matches any ComfyUI type for dependency inputs."""
def __ne__(self, __value: object) -> bool:
return False
any_type = AnyType("*")
class LowVRAMCheckpointLoader(nodes.CheckpointLoaderSimple):
"""Load Checkpoint with dependency input for sequential loading in low VRAM scenarios."""
@classmethod
def INPUT_TYPES(s) -> dict:
base_types = super().INPUT_TYPES()
base_types["optional"] = {
"dependencies": (
any_type,
{
"tooltip": "Connect any output from a previous loader to ensure sequential loading."
},
),
}
return base_types
CATEGORY = "LTXV/loaders"
DESCRIPTION = (
"Loads a diffusion model checkpoint with dependency support. "
"Connect 'dependencies' to a previous loader's output to ensure sequential loading and reduce peak VRAM usage."
)
FUNCTION = "load_checkpoint_sequentially"
def load_checkpoint_sequentially(
self,
ckpt_name: str,
dependencies: object = None, # noqa: ARG002 - required for ComfyUI optional inputs
) -> tuple:
return super().load_checkpoint(ckpt_name)
class LowVRAMAudioVAELoader:
"""Load LTXV Audio VAE with dependency input for sequential loading in low VRAM scenarios."""
@classmethod
def INPUT_TYPES(s) -> dict:
return {
"required": {
"ckpt_name": (
folder_paths.get_filename_list("checkpoints"),
{"tooltip": "Audio VAE checkpoint to load."},
),
},
"optional": {
"dependencies": (
any_type,
{
"tooltip": "Connect any output from a previous loader to ensure sequential loading."
},
),
},
}
RETURN_TYPES = ("VAE",)
RETURN_NAMES = ("audio_vae",)
CATEGORY = "LTXV/loaders"
DESCRIPTION = (
"Loads an LTXV Audio VAE checkpoint with dependency support. "
"Connect 'dependencies' to a previous loader's output to ensure sequential loading and reduce peak VRAM usage."
)
FUNCTION = "load_audio_vae_sequentially"
def load_audio_vae_sequentially(
self,
ckpt_name: str,
dependencies: object = None, # noqa: ARG002 - required for ComfyUI optional inputs
) -> tuple:
ckpt_path = folder_paths.get_full_path_or_raise("checkpoints", ckpt_name)
sd, metadata = comfy.utils.load_torch_file(ckpt_path, return_metadata=True)
audio_vae = AudioVAE(sd, metadata)
return (audio_vae,)
class LowVRAMLatentUpscaleModelLoader(LatentUpscaleModelLoader):
"""Load latent upsampler with dependency input for sequential loading in low VRAM scenarios."""
@classmethod
def define_schema(cls):
return io.Schema(
node_id="LowVRAMLatentUpscaleModelLoader",
display_name="Low VRAMLoad Latent Upscale Model",
category="LTXV/loaders",
inputs=[
io.Combo.Input(
"model_name",
options=folder_paths.get_filename_list("latent_upscale_models"),
),
io.AnyType.Input(
"dependencies",
optional=True,
tooltip="Connect any output from a previous loader to ensure sequential loading.",
),
],
outputs=[
io.LatentUpscaleModel.Output(),
],
)
@classmethod
def execute(cls, model_name, dependencies=None) -> io.NodeOutput:
return LatentUpscaleModelLoader.execute(model_name=model_name)
pos_embedding_exp_values.py
+1922
View File
File diff suppressed because it is too large Load Diff
pyproject.toml
+21 -3
View File
@@ -11,16 +11,22 @@ license = { file = "LICENSE" }
dependencies = [
"diffusers",
"huggingface_hub>=0.25.2",
"transformers[timm]>=4.45.0",
"transformers[timm]>=4.50.0",
"einops",
"ninja~=1.11.1.4"
"ninja~=1.11.1.4",
"accelerate"
]
[project.optional-dependencies]
internal = [
"ltx-video@git+https://github.com/Lightricks/LTX-Video@ltx-video-0.9.1",
"ltx-media-io>=0.1.26",
"av>=10.0.0",
"q8-kernels~=0.1.0"
"q8-kernels~=0.1.4",
"ring_buffer==0.0.9",
"ltx-multigpu-inference~=0.0.13",
"bitsandbytes",
"ultralytics"
]
dev = [
"pre-commit>=4.0.1",
@@ -29,6 +35,18 @@ dev = [
"scikit-image==0.24.0"
]
[tool.uv.sources]
ltx-video@git+https://github.com/Lightricks/LTX-Video@ltx-video-0.9.1",
ring-buffer = {index = "ltx"}
[[tool.uv.index]]
name = "ltx"
url = "https://oauth2accesstoken@us-python.pkg.dev/ltx-artifact-registry/pypi-gar/simple"
default = false
[tool.uv]
keyring-provider = "subprocess"
[tool.isort]
profile = "black"
line_length = 88
q8_nodes.py
+87 -36
View File
@@ -10,7 +10,6 @@ try:
from q8_kernels.functional.ops import hadamard_transform
from q8_kernels.integration.patch_transformer import (
patch_comfyui_native_transformer,
patch_comfyui_transformer,
)
Q8_AVAILABLE = True
@@ -20,6 +19,10 @@ except ImportError:
from .nodes_registry import comfy_node
def list_in_name(check_list, name):
return any([x in name for x in check_list])
def check_q8_available():
if not Q8_AVAILABLE:
raise ImportError(
@@ -39,6 +42,22 @@ class LTXVQ8Patch:
"BOOLEAN",
{"default": False, "tooltip": "Use FP8 attention."},
),
"quantization_preset": (
["0.9.8", "ltxv2", "full_bf16", "custom"],
{"default": "0.9.8"},
),
"quantize_self_attn": (
"BOOLEAN",
{"default": True, "tooltip": "Quantize Self Attention Layer"},
),
"quantize_cross_attn": (
"BOOLEAN",
{"default": True, "tooltip": "Quantize Cross Attention Layer"},
),
"quantize_ffn": (
"BOOLEAN",
{"default": True, "tooltip": "Quantize Feed Forward Layer"},
),
}
}
@@ -46,21 +65,50 @@ class LTXVQ8Patch:
FUNCTION = "patch"
CATEGORY = "lightricks/LTXV"
TITLE = "LTXV Q8 Patcher"
PRESETS = {
"0.9.8": (True, True, True),
"ltxv2": (True, False, True),
"full_bf16": (False, False, False),
}
def patch(self, model, use_fp8_attention):
def patch(
self,
model,
use_fp8_attention,
quantization_preset,
quantize_self_attn,
quantize_cross_attn,
quantize_ffn,
):
check_q8_available()
m = model.clone()
diffusion_key = "diffusion_model"
diffusion_model = m.get_model_object(diffusion_key)
if diffusion_model.__class__.__name__ == "LTXVTransformer3D":
transformer_key = "diffusion_model.transformer"
patcher = patch_comfyui_transformer
else:
transformer_key = "diffusion_model"
patcher = patch_comfyui_native_transformer
transformer_key = "diffusion_model"
transformer = m.get_model_object(transformer_key)
patcher(transformer, use_fp8_attention, True)
m.add_object_patch(transformer_key, transformer)
quantize_self_attn, quantize_cross_attn, quantize_ffn = LTXVQ8Patch.PRESETS.get(
quantization_preset, (quantize_self_attn, quantize_cross_attn, quantize_ffn)
)
if getattr(transformer, "quantization_config", None) is not None:
if (quantize_self_attn, quantize_cross_attn, quantize_ffn) != getattr(
transformer, "quantization_config"
):
quantize_self_attn, quantize_cross_attn, quantize_ffn = getattr(
transformer, "quantization_config"
)
patch_comfyui_native_transformer(
transformer,
use_fp8_attention,
True,
quantize_self_attn,
quantize_cross_attn,
quantize_ffn,
)
setattr(transformer, "is_q8_patched", True)
setattr(
transformer,
"quantization_config",
(quantize_self_attn, quantize_cross_attn, quantize_ffn),
)
setattr(transformer, "use_fp8_attention", use_fp8_attention)
return (m,)
@@ -92,9 +140,9 @@ class LTXVQ8LoraModelLoader:
def load_lora(self, model, lora_name, strength_model):
quant_fn = hadamard_transform
is_patched_transformer = getattr(
model.get_model_object("diffusion_model"), "is_patched", False
)
transformer = model.get_model_object("diffusion_model")
is_patched_transformer = getattr(transformer, "is_q8_patched", False)
if not is_patched_transformer or not Q8_AVAILABLE:
raise ValueError(
"LTXV Q8 Patcher is not applied to the model. Please use LTXQ8Patch node before loading lora or install q8_kernels."
@@ -102,29 +150,32 @@ class LTXVQ8LoraModelLoader:
if strength_model == 0:
return model
quantize_self_attn, quantize_cross_attn, quantize_ffn = getattr(
transformer, "quantization_config"
)
skip_list = []
if not quantize_self_attn:
skip_list += ["attn1"]
if not quantize_cross_attn:
skip_list += ["attn2"]
if not quantize_ffn:
skip_list += ["ff"]
lora_path = folder_paths.get_full_path_or_raise("loras", lora_name)
lora = None
if self.loaded_lora is not None:
if self.loaded_lora[0] == lora_path:
lora = self.loaded_lora[1]
lora = comfy.utils.load_torch_file(lora_path, safe_load=True)
new_lora = {}
for k in lora:
device = lora[k].device
if lora[k].ndim == 2:
if "lora_A" in k and not list_in_name(skip_list, k):
new_lora[k] = quant_fn(
lora[k].to(device="cuda", dtype=torch.bfloat16),
out_type=torch.bfloat16,
).to(device)
else:
new_lora[k] = lora[k]
else:
self.loaded_lora = None
if lora is None:
lora = comfy.utils.load_torch_file(lora_path, safe_load=True)
new_lora = {}
for k in lora:
device = lora[k].device
if lora[k].ndim == 2:
if "lora_A" in k:
new_lora[k] = quant_fn(
lora[k].to(device="cuda", dtype=torch.bfloat16),
out_type=torch.bfloat16,
).to(device)
else:
new_lora[k] = lora[k]
self.loaded_lora = (lora_path, new_lora)
new_lora[k] = lora[k]
self.loaded_lora = (lora_path, new_lora)
model_lora, _ = comfy.sd.load_lora_for_models(
model, None, new_lora, strength_model, 0
requirements.txt
+1 -1
View File
@@ -2,4 +2,4 @@ diffusers
einops
huggingface_hub>=0.25.2
ninja~=1.11.1.4
transformers[timm]>=4.45.0
transformers[timm]>=4.50.0
stg.py
+256 -9
View File
@@ -3,7 +3,7 @@ import json
import math
import os
from dataclasses import dataclass
from typing import List
from typing import List, Optional, Union
import comfy.ldm.modules.attention
import comfy.samplers
@@ -33,6 +33,44 @@ def stg(
return noise_pred
class MomentumBuffer:
def __init__(self, momentum: float):
self.momentum = momentum
self.running_average = 0
def update(self, update_value: torch.Tensor):
new_average = self.momentum * self.running_average
self.running_average = update_value + new_average
def project(v0: torch.Tensor, v1: torch.Tensor):
dtype = v0.dtype
v0, v1 = v0.double(), v1.double()
v1 = torch.nn.functional.normalize(v1, dim=[-1, -2, -3])
v0_parallel = (v0 * v1).sum(dim=[-1, -2, -3], keepdim=True) * v1
v0_orthogonal = v0 - v0_parallel
return v0_parallel.to(dtype), v0_orthogonal.to(dtype)
def apg(
noise_pred_pos: torch.Tensor,
noise_pred_neg: torch.Tensor,
cfg_scale: float,
eta: float = 1.0,
norm_threshold: float = 0.0,
):
diff = noise_pred_pos - noise_pred_neg
if norm_threshold > 0:
ones = torch.ones_like(diff)
diff_norm = diff.norm(p=2, dim=[-1, -2, -3], keepdim=True)
scale_factor = torch.minimum(ones, norm_threshold / diff_norm)
diff = diff * scale_factor
diff_parallel, diff_orthogonal = project(diff, noise_pred_pos)
normalized_update = diff_orthogonal + eta * diff_parallel
noise_pred = noise_pred_pos + (cfg_scale - 1) * normalized_update
return noise_pred
@comfy_node(name="LTXVApplySTG")
class LTXVApplySTG:
@classmethod
@@ -83,15 +121,22 @@ class STGFlag:
# context manager that replaces the attention function in a transformer block
class PatchAttention(contextlib.AbstractContextManager):
def __init__(self, attn_idx=0):
def __init__(self, attn_idx: Optional[Union[int, List[int]]] = None):
self.current_idx = -1
if isinstance(attn_idx, int):
self.attn_idx = [attn_idx]
elif attn_idx is None:
self.attn_idx = [0]
else:
self.attn_idx = list(attn_idx)
def __enter__(self):
self.original_attention = comfy.ldm.modules.attention.optimized_attention
self.original_attention_masked = (
comfy.ldm.modules.attention.optimized_attention_masked
)
self.current_idx = -1
self.attn_idx = attn_idx
def __enter__(self):
comfy.ldm.modules.attention.optimized_attention = self.stg_attention
comfy.ldm.modules.attention.optimized_attention_masked = (
self.stg_attention_masked
@@ -103,16 +148,19 @@ class PatchAttention(contextlib.AbstractContextManager):
self.original_attention_masked
)
self.original_attention = None
self.original_attention_masked = None
def stg_attention(self, q, k, v, heads, *args, **kwargs):
self.current_idx += 1
if self.current_idx == self.attn_idx:
if self.current_idx in self.attn_idx:
return v
else:
return self.original_attention(q, k, v, heads, *args, **kwargs)
def stg_attention_masked(self, q, k, v, heads, *args, **kwargs):
self.current_idx += 1
if self.current_idx == self.attn_idx:
if self.current_idx in self.attn_idx:
return v
else:
return self.original_attention_masked(q, k, v, heads, *args, **kwargs)
@@ -128,8 +176,10 @@ class STGBlockWrapper:
def __call__(self, args, extra_args):
context_manager = contextlib.nullcontext()
stg_indexes = args["transformer_options"].get("stg_indexes", [0])
if self.flag.do_skip and self.idx in self.flag.skip_layers:
context_manager = PatchAttention(0)
context_manager = PatchAttention(stg_indexes)
with context_manager:
hidden_state = extra_args["original_block"](args)
@@ -278,6 +328,10 @@ class STGGuiderAdvanced(comfy.samplers.CFGGuider):
stg_layers_indices_list,
skip_steps_sigma_threshold,
cfg_star_rescale,
apply_apg,
apg_cfg_scale,
eta,
norm_threshold,
):
model = model.clone()
super().__init__(model)
@@ -296,6 +350,10 @@ class STGGuiderAdvanced(comfy.samplers.CFGGuider):
self.stg_layers_indices_list = stg_layers_indices_list
self.skip_steps_sigma_threshold = skip_steps_sigma_threshold
self.cfg_star_rescale = cfg_star_rescale
self.apply_apg = apply_apg
self.apg_cfg_scale = apg_cfg_scale
self.eta = eta
self.norm_threshold = norm_threshold
STGGuider.patch_model(model, self.stg_flag)
def sigma_to_params_mapping(self, sigma):
@@ -351,7 +409,9 @@ class STGGuiderAdvanced(comfy.samplers.CFGGuider):
noise_pred_neg = 0
noise_pred_perturbed = 0
if not math.isclose(cfg_value, 1.0):
if not math.isclose(cfg_value, 1.0) or (
self.apply_apg and not math.isclose(self.apg_cfg_scale, 1.0)
):
noise_pred_neg = comfy.samplers.calc_cond_batch(
self.inner_model,
[negative_cond],
@@ -396,6 +456,16 @@ class STGGuiderAdvanced(comfy.samplers.CFGGuider):
stg_rescale,
)
if self.apply_apg:
stg_result = apg(
stg_result,
noise_pred_neg,
cfg_scale=self.apg_cfg_scale,
momentum_buffer=None,
eta=self.eta,
norm_threshold=self.norm_threshold,
)
# normally this would be done in cfg_function, but we skipped
# that for efficiency: we can compute the noise predictions in
# a single call to calc_cond_batch() (rather than two)
@@ -553,6 +623,25 @@ class STGGuiderAdvancedNode:
"tooltip": "Preset resolution and frame count. Custom allows manual input.",
},
),
"apply_apg": (
"BOOLEAN",
{
"default": False,
"tooltip": "If true, applies the APG (Adaptive Projections Guidance) to the STG.",
},
),
"apg_cfg_scale": (
"FLOAT",
{"default": 1.0, "min": 0.0, "max": 100.0, "step": 0.01},
),
"eta": (
"FLOAT",
{"default": 1.0, "min": 0.0, "max": 100.0, "step": 0.01},
),
"norm_threshold": (
"FLOAT",
{"default": 0.0, "min": 0.0, "max": 100.0, "step": 0.01},
),
},
}
@@ -626,6 +715,10 @@ class STGGuiderAdvancedNode:
stg_rescale_values,
stg_layers_indices,
preset=None,
apply_apg=False,
apg_cfg_scale=1.0,
eta=1.0,
norm_threshold=0.0,
):
if preset and preset != "Custom":
preset_data = next(
@@ -666,6 +759,10 @@ class STGGuiderAdvancedNode:
stg_layers_indices_list,
skip_steps_sigma_threshold,
cfg_star_rescale,
apply_apg,
apg_cfg_scale=apg_cfg_scale,
eta=eta,
norm_threshold=norm_threshold,
)
guider.set_conds(positive, negative)
guider.raw_conds = (positive, negative)
@@ -695,3 +792,153 @@ class STGAdvancedPresetsNode:
def get_preset(self, preset=None):
return (preset,)
class APGGuider(comfy.samplers.CFGGuider):
def __init__(
self,
model: ModelPatcher,
cfg_scale,
eta,
norm_threshold,
momentum_coefficient: float = -0.9,
):
self.model = model.clone()
super().__init__(self.model)
self.momentum_coefficient = momentum_coefficient
self.momentum_buffer = MomentumBuffer(self.momentum_coefficient)
self.eta = eta
self.norm_threshold = norm_threshold
self.cfg_scale = cfg_scale
self.previous_timestep = None
def set_conds(self, positive, negative):
self.inner_set_conds({"positive": positive, "negative": negative})
def predict_noise(
self,
x: torch.Tensor,
timestep: torch.Tensor,
model_options: dict = {},
seed=None,
):
# CFG zero init - skipping steps with timestep bigger than given threshold.
# in CFGGuider.predict_noise, we call sampling_function(), which uses cfg_function() to compute pos & neg
# but we'd rather do a single batch of sampling pos, neg, and perturbed, so we call calc_cond_batch([perturbed,pos,neg]) directly
positive_cond = self.conds.get("positive", None)
negative_cond = self.conds.get("negative", None)
noise_pred_pos = comfy.samplers.calc_cond_batch(
self.inner_model,
[positive_cond],
x,
timestep,
model_options,
)[0]
if (
self.previous_timestep is not None
and timestep.item() > self.previous_timestep
):
print("Resetting momentum buffer")
self.momentum_buffer = MomentumBuffer(self.momentum_coefficient)
noise_pred_neg = 0
if not math.isclose(self.cfg_scale, 1.0):
noise_pred_neg = comfy.samplers.calc_cond_batch(
self.inner_model,
[negative_cond],
x,
timestep,
model_options,
)[0]
apg_result = apg(
noise_pred_pos,
noise_pred_neg,
self.cfg_scale,
self.eta,
self.norm_threshold,
)
# normally this would be done in cfg_function, but we skipped
# that for efficiency: we can compute the noise predictions in
# a single call to calc_cond_batch() (rather than two)
# so we replicate the hook here
for fn in model_options.get("sampler_post_cfg_function", []):
args = {
"denoised": apg_result,
"cond": positive_cond,
"uncond": negative_cond,
"model": self.inner_model,
"uncond_denoised": noise_pred_neg,
"cond_denoised": noise_pred_pos,
"sigma": timestep,
"model_options": model_options,
"input": x,
}
apg_result = fn(args)
self.previous_timestep = timestep.item()
return apg_result
@comfy_node(name="APGGuider")
class APGGuiderNode:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"model": ("MODEL",),
"positive": ("CONDITIONING",),
"negative": ("CONDITIONING",),
"cfg_scale": (
"FLOAT",
{"default": 1.0, "min": 0.0, "max": 100.0, "step": 0.01},
),
"eta": (
"FLOAT",
{"default": 1.0, "min": 0.0, "max": 100.0, "step": 0.01},
),
"norm_threshold": (
"FLOAT",
{"default": 0.0, "min": 0.0, "max": 100.0, "step": 0.01},
),
"momentum_coefficient": (
"FLOAT",
{"default": -0.9, "min": -3.0, "max": 3.0, "step": 0.01},
),
},
}
RETURN_TYPES = ("GUIDER",)
FUNCTION = "get_guider"
CATEGORY = "lightricks/LTXV"
DESCRIPTION = """
The APG Guider implements Adaptive Projected Guidance (APG).
Reference: https://arxiv.org/abs/2410.02416.
"""
def get_guider(
self,
model,
positive,
negative,
cfg_scale,
eta,
norm_threshold,
momentum_coefficient,
):
guider = APGGuider(
model,
cfg_scale,
eta,
norm_threshold,
momentum_coefficient,
)
guider.set_conds(positive, negative)
guider.raw_conds = (positive, negative)
return (guider,)
system_prompts/gemma_i2v_system_prompt.txt
+30
View File
@@ -0,0 +1,30 @@
You are a Creative Assistant writing concise, action-focused image-to-video prompts. Given an image (first frame) and user Raw Input Prompt, generate a prompt to guide video generation from that image.
#### Guidelines:
- Analyze the Image: Identify Subject, Setting, Elements, Style and Mood.
- Follow user Raw Input Prompt: Include all requested motion, actions, camera movements, audio, and details. If in conflict with the image, prioritize user request while maintaining visual consistency (describe transition from image to user's scene).
- Describe only changes from the image: Don't reiterate established visual details. Inaccurate descriptions may cause scene cuts.
- Active language: Use present-progressive verbs ("is walking," "speaking"). If no action specified, describe natural movements.
- Chronological flow: Use temporal connectors ("as," "then," "while").
- Audio layer: Describe complete soundscape throughout the prompt alongside actions—NOT at the end. Align audio intensity with action tempo. Include natural background audio, ambient sounds, effects, speech or music (when requested). Be specific (e.g., "soft footsteps on tile") not vague (e.g., "ambient sound").
- Speech (only when requested): Provide exact words in quotes with character's visual/voice characteristics (e.g., "The tall man speaks in a low, gravelly voice"), language if not English and accent if relevant. If general conversation mentioned without text, generate contextual quoted dialogue. (i.e., "The man is talking" input -> the output should include exact spoken words, like: "The man is talking in an excited voice saying: 'You won't believe what I just saw!' His hands gesture expressively as he speaks, eyebrows raised with enthusiasm. The ambient sound of a quiet room underscores his animated speech.")
- Style: Include visual style at beginning: "Style: <style>, <rest of prompt>." If unclear, omit to avoid conflicts.
- Visual and audio only: Describe only what is seen and heard. NO smell, taste, or tactile sensations.
- Restrained language: Avoid dramatic terms. Use mild, natural, understated phrasing.
#### Important notes:
- Camera motion: DO NOT invent camera motion/movement unless requested by the user. Make sure to include camera motion only if specified in the input.
- Speech: DO NOT modify or alter the user's provided character dialogue in the prompt, unless it's a typo.
- No timestamps or cuts: DO NOT use timestamps or describe scene cuts unless explicitly requested.
- Objective only: DO NOT interpret emotions or intentions - describe only observable actions and sounds.
- Format: DO NOT use phrases like "The scene opens with..." / "The video starts...". Start directly with Style (optional) and chronological scene description.
- Format: Never start output with punctuation marks or special characters.
- DO NOT invent dialogue unless the user mentions speech/talking/singing/conversation.
- Your performance is CRITICAL. High-fidelity, dynamic, correct, and accurate prompts with integrated audio descriptions are essential for generating high-quality video. Your goal is flawless execution of these rules.
#### Output Format (Strict):
- Single concise paragraph in natural English. NO titles, headings, prefaces, sections, code fences, or Markdown.
- If unsafe/invalid, return original user prompt. Never ask questions or clarifications.
#### Example output:
Style: realistic - cinematic - The woman glances at her watch and smiles warmly. She speaks in a cheerful, friendly voice, "I think we're right on time!" In the background, a café barista prepares drinks at the counter. The barista calls out in a clear, upbeat tone, "Two cappuccinos ready!" The sound of the espresso machine hissing softly blends with gentle background chatter and the light clinking of cups on saucers.
system_prompts/gemma_t2v_system_prompt.txt
+40
View File
@@ -0,0 +1,40 @@
You are a Creative Assistant. Given a user's raw input prompt describing a scene or concept, expand it into a detailed video generation prompt with specific visuals and integrated audio to guide a text-to-video model.
#### Guidelines
- Strictly follow all aspects of the user's raw input: include every element requested (style, visuals, motions, actions, camera movement, audio).
- If the input is vague, invent concrete details: lighting, textures, materials, scene settings, etc.
- For characters: describe gender, clothing, hair, expressions. DO NOT invent unrequested characters.
- Use active language: present-progressive verbs ("is walking," "speaking"). If no action specified, describe natural movements.
- Maintain chronological flow: use temporal connectors ("as," "then," "while").
- Audio layer: Describe complete soundscape (background audio, ambient sounds, SFX, speech/music when requested). Integrate sounds chronologically alongside actions. Be specific (e.g., "soft footsteps on tile"), not vague (e.g., "ambient sound is present").
- Speech (only when requested):
- For ANY speech-related input (talking, conversation, singing, etc.), ALWAYS include exact words in quotes with voice characteristics (e.g., "The man says in an excited voice: 'You won't believe what I just saw!'").
- Specify language if not English and accent if relevant.
- Style: Include visual style at the beginning: "Style: <style>, <rest of prompt>." Default to cinematic-realistic if unspecified. Omit if unclear.
- Visual and audio only: NO non-visual/auditory senses (smell, taste, touch).
- Restrained language: Avoid dramatic/exaggerated terms. Use mild, natural phrasing.
- Colors: Use plain terms ("red dress"), not intensified ("vibrant blue," "bright red").
- Lighting: Use neutral descriptions ("soft overhead light"), not harsh ("blinding light").
- Facial features: Use delicate modifiers for subtle features (i.e., "subtle freckles").
#### Important notes:
- Analyze the user's raw input carefully. In cases of FPV or POV, exclude the description of the subject whose POV is requested.
- Camera motion: DO NOT invent camera motion unless requested by the user.
- Speech: DO NOT modify user-provided character dialogue unless it's a typo.
- No timestamps or cuts: DO NOT use timestamps or describe scene cuts unless explicitly requested.
- Format: DO NOT use phrases like "The scene opens with...". Start directly with Style (optional) and chronological scene description.
- Format: DO NOT start your response with special characters.
- DO NOT invent dialogue unless the user mentions speech/talking/singing/conversation.
- If the user's raw input prompt is highly detailed, chronological and in the requested format: DO NOT make major edits or introduce new elements. Add/enhance audio descriptions if missing.
#### Output Format (Strict):
- Single continuous paragraph in natural language (English).
- NO titles, headings, prefaces, code fences, or Markdown.
- If unsafe/invalid, return original user prompt. Never ask questions or clarifications.
Your output quality is CRITICAL. Generate visually rich, dynamic prompts with integrated audio for high-quality video generation.
#### Example
Input: "A woman at a coffee shop talking on the phone"
Output:
Style: realistic with cinematic lighting. In a medium close-up, a woman in her early 30s with shoulder-length brown hair sits at a small wooden table by the window. She wears a cream-colored turtleneck sweater, holding a white ceramic coffee cup in one hand and a smartphone to her ear with the other. Ambient cafe sounds fill the space—espresso machine hiss, quiet conversations, gentle clinking of cups. The woman listens intently, nodding slightly, then takes a sip of her coffee and sets it down with a soft clink. Her face brightens into a warm smile as she speaks in a clear, friendly voice, 'That sounds perfect! I'd love to meet up this weekend. How about Saturday afternoon?' She laughs softly—a genuine chuckle—and shifts in her chair. Behind her, other patrons move subtly in and out of focus. 'Great, I'll see you then,' she concludes cheerfully, lowering the phone.
tiled_sampler.py
+6 -2
View File
@@ -205,7 +205,7 @@ class LTXVTiledSampler:
tile_positive,
tile_negative,
tile_latents,
) = LTXVAddGuide().generate(
) = LTXVAddGuide().execute(
positive=tile_positive,
negative=tile_negative,
vae=vae,
@@ -273,7 +273,11 @@ class LTXVTiledSampler:
# Clean up guides if image conditioning was used
if cond_images is not None:
tile_positive, tile_negative, denoised_tile = LTXVCropGuides().crop(
(
tile_positive,
tile_negative,
denoised_tile,
) = LTXVCropGuides().execute(
positive=tile_positive,
negative=tile_negative,
latent=denoised_tile,
tiled_vae_decode.py
+305 -25
View File
@@ -1,3 +1,5 @@
import logging
import torch
from .nodes_registry import comfy_node
@@ -19,6 +21,10 @@ class LTXVTiledVAEDecode:
"overlap": ("INT", {"default": 1, "min": 1, "max": 8}),
"last_frame_fix": ("BOOLEAN", {"default": False}),
},
"optional": {
"working_device": (["cpu", "auto"], {"default": "auto"}),
"working_dtype": (["float16", "float32", "auto"], {"default": "auto"}),
},
}
RETURN_TYPES = ("IMAGE",)
@@ -36,6 +42,8 @@ class LTXVTiledVAEDecode:
vertical_tiles,
overlap,
last_frame_fix,
working_device="auto",
working_dtype="auto",
):
# Get the latent samples
samples = latents["samples"]
@@ -67,6 +75,31 @@ class LTXVTiledVAEDecode:
output = None
weights = None
target_device = samples.device if working_device == "auto" else working_device
if working_dtype == "auto":
target_dtype = samples.dtype
elif working_dtype == "float16":
target_dtype = torch.float16
elif working_dtype == "float32":
target_dtype = torch.float32
output = torch.zeros(
(
batch,
image_frames,
output_height,
output_width,
3,
),
device=target_device,
dtype=target_dtype,
)
weights = torch.zeros(
(batch, image_frames, output_height, output_width, 1),
device=target_device,
dtype=target_dtype,
)
# Process each tile
for v in range(vertical_tiles):
for h in range(horizontal_tiles):
@@ -90,9 +123,9 @@ class LTXVTiledVAEDecode:
tile_height = v_end - v_start
tile_width = h_end - h_start
print(f"Processing VAE decode tile at row {v}, col {h}:")
print(f" Position: ({v_start}:{v_end}, {h_start}:{h_end})")
print(f" Size: {tile_height}x{tile_width}")
logging.info(f"Processing VAE decode tile at row {v}, col {h}:")
logging.info(f" Position: ({v_start}:{v_end}, {h_start}:{h_end})")
logging.info(f" Size: {tile_height}x{tile_width}")
# Extract tile
tile = samples[:, :, :, v_start:v_end, h_start:h_end]
@@ -103,25 +136,6 @@ class LTXVTiledVAEDecode:
# Decode the tile
decoded_tile = vae.decode(tile_latents["samples"])
# Initialize output tensors on first tile
if output is None:
output = torch.zeros(
(
batch,
image_frames,
output_height,
output_width,
decoded_tile.shape[-1],
),
device=decoded_tile.device,
dtype=decoded_tile.dtype,
)
weights = torch.zeros(
(batch, image_frames, output_height, output_width, 1),
device=decoded_tile.device,
dtype=decoded_tile.dtype,
)
# Calculate output tile boundaries
out_h_start = v_start * height_scale_factor
out_h_end = v_end * height_scale_factor
@@ -176,15 +190,15 @@ class LTXVTiledVAEDecode:
# Add weighted tile to output
output[:, :, out_h_start:out_h_end, out_w_start:out_w_end, :] += (
decoded_tile * tile_weights
)
).to(target_device, target_dtype)
# Add weights to weight tensor
weights[
:, :, out_h_start:out_h_end, out_w_start:out_w_end, :
] += tile_weights
] += tile_weights.to(target_device, target_dtype)
# Normalize by weights
output = output / (weights + 1e-8)
output /= weights + 1e-8
# Reshape output to match expected format [batch * frames, height, width, channels]
output = output.view(
@@ -195,3 +209,269 @@ class LTXVTiledVAEDecode:
output = output[:-time_scale_factor, :, :]
return (output,)
def compute_chunk_boundaries(
chunk_start: int,
temporal_tile_length: int,
temporal_overlap: int,
total_latent_frames: int,
):
"""Compute chunk boundaries for temporal tiling.
Args:
chunk_start: Starting frame index for the current chunk
temporal_tile_length: Length of each temporal tile
temporal_overlap: Number of frames to overlap between chunks
total_latent_frames: Total number of latent frames
Returns:
Tuple of (overlap_start, chunk_end)
"""
if chunk_start == 0:
# First chunk: no overlap needed
chunk_end = min(chunk_start + temporal_tile_length, total_latent_frames)
overlap_start = chunk_start
else:
# Subsequent chunks: include overlap from previous chunk
# -1 because we need one extra frame to overlap, which is decoded to a single frame
# never overlap with the first latent frame
overlap_start = max(1, chunk_start - temporal_overlap - 1)
extra_frames = chunk_start - overlap_start
chunk_end = min(
chunk_start + temporal_tile_length - extra_frames,
total_latent_frames,
)
return overlap_start, chunk_end
def calculate_temporal_output_boundaries(
overlap_start: int, time_scale_factor: int, tile_out_frames: int
):
"""Calculate temporal output boundaries for the decoded tile.
Args:
overlap_start: Starting frame index including overlap
time_scale_factor: Time scaling factor from VAE
tile_out_frames: Number of frames in the decoded tile
Returns:
Tuple of (out_t_start, out_t_end)
"""
# +1 for the first frame
out_t_start = 1 + overlap_start * time_scale_factor
# Calculate actual output temporal dimensions
out_t_end = out_t_start + tile_out_frames
return out_t_start, out_t_end
@comfy_node(
name="LTXVSpatioTemporalTiledVAEDecode",
)
class LTXVSpatioTemporalTiledVAEDecode(LTXVTiledVAEDecode):
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"vae": ("VAE", {"tooltip": "The VAE to use."}),
"latents": ("LATENT", {"tooltip": "The latent samples to decode."}),
"spatial_tiles": (
"INT",
{
"default": 4,
"min": 1,
"max": 8,
"tooltip": "The number of spatial tiles to use, horizontal and vertical.",
},
),
"spatial_overlap": (
"INT",
{
"default": 1,
"min": 0,
"max": 8,
"tooltip": "The overlap between the spatial tiles. (in latent frames)",
},
),
"temporal_tile_length": (
"INT",
{
"default": 16,
"min": 2,
"max": 1000,
"tooltip": "The length of the temporal tile to use for the sampling, in latent frames, including the overlapping region.",
},
),
"temporal_overlap": (
"INT",
{
"default": 1,
"min": 0,
"max": 8,
"tooltip": "The overlap between the temporal tiles, in latent frames.",
},
),
"last_frame_fix": (
"BOOLEAN",
{
"default": False,
"tooltip": "If true, the last frame will be repeated and discarded after the decoding.",
},
),
"working_device": (
["cpu", "auto"],
{
"default": "auto",
"tooltip": "The device to use for the decoding. auto->same as the latents.",
},
),
"working_dtype": (
["float16", "float32", "auto"],
{
"default": "auto",
"tooltip": "The data type to use for the decoding. auto->same as the latents.",
},
),
},
}
RETURN_TYPES = ("IMAGE",)
RETURN_NAMES = ("image",)
FUNCTION = "decode_spatial_temporal"
CATEGORY = "latent"
def decode_spatial_temporal(
self,
vae,
latents,
spatial_tiles=4,
spatial_overlap=1,
temporal_tile_length=16,
temporal_overlap=1,
last_frame_fix=False,
working_device="auto",
working_dtype="auto",
):
if temporal_tile_length < temporal_overlap + 1:
raise ValueError(
"Temporal tile length must be greater than temporal overlap + 1"
)
# Get the latent samples
samples = latents["samples"]
batch, channels, frames, height, width = samples.shape
time_scale_factor, width_scale_factor, height_scale_factor = (
vae.downscale_index_formula
)
image_frames = 1 + (frames - 1) * time_scale_factor
# Calculate output image dimensions
output_height = height * height_scale_factor
output_width = width * width_scale_factor
target_device = samples.device if working_device == "auto" else working_device
if working_dtype == "auto":
target_dtype = samples.dtype
elif working_dtype == "float16":
target_dtype = torch.float16
elif working_dtype == "float32":
target_dtype = torch.float32
# Initialize output tensor and weight tensor
output = torch.empty(
(
batch,
image_frames,
output_height,
output_width,
3,
),
device=target_device,
dtype=target_dtype,
)
# Process temporal chunks similar to reference function
total_latent_frames = frames
chunk_start = 0
while chunk_start < total_latent_frames:
# Calculate chunk boundaries
overlap_start, chunk_end = compute_chunk_boundaries(
chunk_start, temporal_tile_length, temporal_overlap, total_latent_frames
)
# units are latent frames
chunk_frames = chunk_end - overlap_start
logging.info(
f"Processing temporal chunk: {overlap_start}:{chunk_end} ({chunk_frames} latent frames)"
)
# Extract tile
tile = samples[:, :, overlap_start:chunk_end]
# Create tile latents dict
tile_latents = {"samples": tile}
# Decode the tile
decoded_tile = self.decode(
vae=vae,
latents=tile_latents,
vertical_tiles=spatial_tiles,
horizontal_tiles=spatial_tiles,
overlap=spatial_overlap,
last_frame_fix=last_frame_fix,
working_device=working_device,
working_dtype=working_dtype,
)[0][None]
if chunk_start == 0:
output[:, : decoded_tile.shape[1]] = decoded_tile
# Drop first frame if needed (overlap)
else:
if decoded_tile.shape[1] == 1:
raise ValueError("Dropping first frame but tile has only 1 frame")
decoded_tile = decoded_tile[:, 1:] # Drop first frame
# Calculate temporal output boundaries
out_t_start, out_t_end = calculate_temporal_output_boundaries(
overlap_start, time_scale_factor, decoded_tile.shape[1]
)
# Create weight mask for this tile
overlap_frames = temporal_overlap * time_scale_factor
frame_weights = torch.linspace(
0,
1,
overlap_frames + 2,
device=decoded_tile.device,
dtype=decoded_tile.dtype,
)[1:-1]
tile_weights = frame_weights.view(1, -1, 1, 1, 1)
after_overlap_frames_start = out_t_start + overlap_frames
# Add weighted tile to output
overlap_output = decoded_tile[:, :overlap_frames]
output[:, out_t_start:after_overlap_frames_start] *= 1 - tile_weights
output[:, out_t_start:after_overlap_frames_start] += (
tile_weights * overlap_output
)
output[:, after_overlap_frames_start:out_t_end] = decoded_tile[
:, overlap_frames:
]
# Move to next chunk
chunk_start = chunk_end
# Reshape output to match expected format [batch * frames, height, width, channels]
output = output.view(
batch * image_frames, output_height, output_width, output.shape[-1]
)
return (output,)
tricks/modules/ltx_model.py
+1 -2
View File
@@ -7,7 +7,6 @@ from comfy.ldm.lightricks.model import (
BasicTransformerBlock,
LTXVModel,
apply_rotary_emb,
precompute_freqs_cis,
)
from comfy.ldm.lightricks.symmetric_patchifier import latent_to_pixel_coords
from torch import nn
@@ -167,7 +166,7 @@ class LTXVModelModified(LTXVModel):
(attention_mask.shape[0], 1, -1, attention_mask.shape[-1])
) * torch.finfo(x.dtype).max
pe = precompute_freqs_cis(
pe = self._precompute_freqs_cis(
fractional_coords, dim=self.inner_dim, out_dtype=x.dtype
)
utiltily_nodes.py
+34
View File
@@ -1,5 +1,39 @@
from .nodes_registry import comfy_node
# Internal keys used to store AV merge metadata inside model options
_AV_META_KEY = "ltxv_av_meta"
_AV_META_VIDEO_ONLY_KEYS = "video_only_keys"
_AV_META_AUDIO_ONLY_KEYS = "audio_only_keys"
_AV_META_SHARED_KEYS = "shared_keys"
_AV_META_AUDIO_SHARED_VALUES = "audio_shared_values"
# Common key constants
_KEY_SAMPLES = "samples"
_KEY_TYPE = "type"
_OPT_TRANSFORMER = "transformer_options"
_OPT_AUDIO_LENGTH = "audio_length"
def _build_av_meta(video_latent: dict, audio_latent: dict) -> dict:
"""Compute metadata that enables lossless split after AV concat.
Notes:
- We keep video values for shared keys in the merged latent, so we must
persist the original audio values per shared key.
"""
video_keys = set(video_latent.keys()) - {_KEY_SAMPLES}
audio_keys = set(audio_latent.keys()) - {_KEY_SAMPLES, _KEY_TYPE}
video_only_keys = list(video_keys - audio_keys)
audio_only_keys = list(audio_keys - video_keys)
shared_keys = list(video_keys & audio_keys)
audio_shared_values = {k: audio_latent[k] for k in shared_keys}
return {
_AV_META_VIDEO_ONLY_KEYS: video_only_keys,
_AV_META_AUDIO_ONLY_KEYS: audio_only_keys,
_AV_META_SHARED_KEYS: shared_keys,
_AV_META_AUDIO_SHARED_VALUES: audio_shared_values,
}
@comfy_node(description="Image to CPU")
class ImageToCPU:
vae_patcher.py
+1 -1
View File
@@ -19,7 +19,7 @@ class LTXVPatcherVAE:
TITLE = "LTXV VAE Patcher"
def patch(self, vae):
from .vae_patcher.patch import patch_vae
from q8_kernels.integration.patch_vae import patch_vae
vae_model = vae.first_stage_model
vae.memory_used_decode = (
vae_patcher/__init__.py
View File
vae_patcher/forwards.py
-273
View File
@@ -1,273 +0,0 @@
from typing import Optional
import numpy as np
import torch
from einops import rearrange
from .kernels.ops import add_inplace, pixel_norm_inplace
def find_split_size(F: int) -> int:
s = 16
while F % s < 3:
s -= 1
return s
def inplace_patch_conv_2(x, conv: torch.nn.Conv3d):
assert conv.kernel_size == (3, 3, 3)
assert conv.stride == (1, 1, 1)
assert conv.padding == (0, 1, 1)
assert conv.dilation == (1, 1, 1)
assert conv.groups == 1
x[:, :, 0, :, :] = x[:, :, 1, :, :].clone()
x[:, :, -1, :, :] = x[:, :, -2, :, :].clone()
if x.shape[2] > 16:
split_size = find_split_size(x.shape[2])
num_splits = (x.shape[2] + split_size - 1) // split_size
else:
split_size = x.shape[2] - 1
num_splits = 1
out_channels = conv.out_channels
in_channels = conv.in_channels
x_buffers = torch.empty(
x.shape[0],
x.shape[1],
1,
x.shape[3],
x.shape[4],
device=x.device,
dtype=x.dtype,
)
o_buffers = torch.empty(
x.shape[0],
x.shape[1],
1,
x.shape[3],
x.shape[4],
device=x.device,
dtype=x.dtype,
)
# 0 case
x_buffers[:, :, 0, :, :] = x[:, :, split_size - 1, :, :].clone()
x[:, :out_channels, 1:split_size, :, :] = conv(
x[:, :in_channels, : split_size + 1, :, :]
)
for i in range(1, num_splits):
curr_frame_start = i * split_size
curr_frame_end = min((i + 1) * split_size, x.shape[2] - 1)
o_buffers[:, :, 0, :, :] = x[:, :, curr_frame_start - 1, :, :].clone()
x[:, :, curr_frame_start - 1, :, :] = x_buffers[:, :, 0, :, :].clone()
x_buffers[:, :, 0, :, :] = x[:, :, curr_frame_end - 1, :, :].clone()
x[:, :out_channels, curr_frame_start:curr_frame_end, :, :] = conv(
x[:, :in_channels, curr_frame_start - 1 : curr_frame_end + 1, :, :]
)
x[:, :, curr_frame_start - 1, :, :] = o_buffers[:, :, 0, :, :].clone()
def res_block_forward(
self,
hidden_states: torch.FloatTensor,
causal: bool = True,
timestep: Optional[torch.Tensor] = None,
) -> torch.FloatTensor:
batch_size = hidden_states.shape[0]
ada_values = self.scale_shift_table[None, ..., None, None, None].to(
timestep.device
) + timestep.reshape(
batch_size,
4,
-1,
timestep.shape[-3],
timestep.shape[-2],
timestep.shape[-1],
)
shift1, scale1, shift2, scale2 = ada_values.unbind(dim=1)
pixel_norm_inplace(hidden_states, scale1, shift1, 1e-9)
inplace_patch_conv_2(hidden_states, self.conv1.conv)
pixel_norm_inplace(hidden_states, scale2, shift2, 1e-9)
inplace_patch_conv_2(hidden_states, self.conv2.conv)
def block_forward(
self,
hidden_states: torch.FloatTensor,
causal: bool = True,
timestep: Optional[torch.Tensor] = None,
) -> torch.FloatTensor:
timestep_embed = None
if self.timestep_conditioning:
assert (
timestep is not None
), "should pass timestep with timestep_conditioning=True"
batch_size = hidden_states.shape[0]
timestep_embed = self.time_embedder(
timestep=timestep.flatten(),
resolution=None,
aspect_ratio=None,
batch_size=batch_size,
hidden_dtype=hidden_states.dtype,
)
timestep_embed = timestep_embed.view(
batch_size, timestep_embed.shape[-1], 1, 1, 1
)
workspace = torch.empty(
hidden_states.shape[0],
hidden_states.shape[1],
hidden_states.shape[2] + 2,
hidden_states.shape[3],
hidden_states.shape[4],
device=hidden_states.device,
dtype=hidden_states.dtype,
memory_format=torch.channels_last_3d,
)
for resnet in self.res_blocks:
workspace[:, :, 1:-1].copy_(hidden_states)
resnet(workspace, causal=causal, timestep=timestep_embed)
add_inplace(hidden_states, workspace, 1)
del workspace
torch.cuda.empty_cache()
return hidden_states
def unpatchify(x, patch_size_hw, patch_size_t=1):
if patch_size_hw == 1 and patch_size_t == 1:
return x
if x.dim() == 4:
x = rearrange(
x, "b (c r q) h w -> b c (h q) (w r)", q=patch_size_hw, r=patch_size_hw
)
elif x.dim() == 5:
x = rearrange(
x,
"b (c p r q) f h w -> b c (f p) (h q) (w r)",
p=patch_size_t,
q=patch_size_hw,
r=patch_size_hw,
)
return x
def upsample_forward(self, x, causal: bool = True):
if self.residual:
# Reshape and duplicate the input to match the output shape
x_in = rearrange(
x,
"b (c p1 p2 p3) d h w -> b c (d p1) (h p2) (w p3)",
p1=self.stride[0],
p2=self.stride[1],
p3=self.stride[2],
)
num_repeat = np.prod(self.stride) // self.out_channels_reduction_factor
x_in = x_in.repeat(1, num_repeat, 1, 1, 1)
workspace = torch.empty(
x.shape[0],
self.conv.conv.out_channels,
x.shape[2] + 2,
x.shape[3],
x.shape[4],
device=x.device,
dtype=x.dtype,
)
workspace[:, : x.shape[1], 1:-1, :, :].copy_(x)
del x
torch.cuda.empty_cache()
inplace_patch_conv_2(workspace, self.conv.conv)
x = workspace[:, : self.conv.conv.out_channels, 1:-1, :, :]
x = rearrange(
x,
"b (c p1 p2 p3) d h w -> b c (d p1) (h p2) (w p3)",
p1=self.stride[0],
p2=self.stride[1],
p3=self.stride[2],
)
if self.residual:
x.add_(x_in)
del x_in
torch.cuda.empty_cache()
return x[:, :, 1:, :, :]
def decoder_forward(
self,
sample: torch.FloatTensor,
timestep: Optional[torch.Tensor] = None,
) -> torch.FloatTensor:
r"""The forward method of the `Decoder` class."""
batch_size = sample.shape[0]
sample = self.conv_in(sample, causal=self.causal)
upscale_dtype = next(iter(self.up_blocks.parameters())).dtype
sample = sample.to(upscale_dtype)
if self.timestep_conditioning:
assert (
timestep is not None
), "should pass timestep with timestep_conditioning=True"
scaled_timestep = (timestep * self.timestep_scale_multiplier).to(sample.device)
for up_block in self.up_blocks:
if (
self.timestep_conditioning
and up_block.__class__.__name__ == "UNetMidBlock3D"
):
sample = up_block(sample, causal=self.causal, timestep=scaled_timestep)
else:
sample = up_block(sample, causal=self.causal)
if self.timestep_conditioning:
embedded_timestep = self.last_time_embedder(
timestep=scaled_timestep.flatten(),
resolution=None,
aspect_ratio=None,
batch_size=sample.shape[0],
hidden_dtype=sample.dtype,
)
embedded_timestep = embedded_timestep.view(
batch_size, embedded_timestep.shape[-1], 1, 1, 1
)
ada_values = self.last_scale_shift_table[None, ..., None, None, None].to(
embedded_timestep.device
) + embedded_timestep.reshape(
batch_size,
2,
-1,
embedded_timestep.shape[-3],
embedded_timestep.shape[-2],
embedded_timestep.shape[-1],
)
shift, scale = ada_values.unbind(dim=1)
workspace = torch.empty(
sample.shape[0],
sample.shape[1],
sample.shape[2] + 2,
sample.shape[3],
sample.shape[4],
device=sample.device,
dtype=sample.dtype,
memory_format=torch.channels_last_3d,
)
workspace[:, :, 1:-1, :, :].copy_(sample)
del sample
torch.cuda.empty_cache()
pixel_norm_inplace(workspace, scale, shift, 1e-9)
inplace_patch_conv_2(workspace, self.conv_out.conv)
return unpatchify(
workspace[:, : self.conv_out.conv.out_channels, 1:-1],
patch_size_hw=self.patch_size,
patch_size_t=1,
)
vae_patcher/kernels/__init__.py
View File
vae_patcher/kernels/add_inplace.cpp
-31
View File
@@ -1,31 +0,0 @@
#include <ATen/cuda/CUDAContext.h>
#include <c10/cuda/CUDAGuard.h>
#include <torch/extension.h>
#include <torch/python.h>
#include <vector>
void inplace_add_cuda(
at::Tensor& x,
at::Tensor& input2,
int offset,
cudaStream_t stream
);
void inplace_add(at::Tensor &x, at::Tensor &workspace, int offset) {
at::cuda::CUDAGuard device_guard{x.get_device()};
auto stream = at::cuda::getCurrentCUDAStream().stream();
inplace_add_cuda(
x,
workspace,
offset,
stream
);
}
PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
m.def("inplace_add", &inplace_add,
"inplace_add");
}
vae_patcher/kernels/add_inplace_cuda.cu
-91
View File
@@ -1,91 +0,0 @@
#include <c10/util/BFloat16.h>
#include <c10/cuda/CUDAException.h> // For C10_CUDA_CHECK and C10_CUDA_KERNEL_LAUNCH_CHECK
#include <cuda_runtime.h>
#include <cuda_bf16.h>
#include <stdio.h>
#include <limits>
#include <cuda.h>
#include <ATen/ATen.h>
#include <torch/types.h>
using bfloat16 = at::BFloat16;
using accessor = at::PackedTensorAccessor64<at::BFloat16, 5>;
constexpr int BLOCK_ROWS = 8;
template <typename accessor_t>
__global__ void add_tensors_tiled(const accessor_t tensor1,
const accessor_t tensor2,
accessor_t result,
int offset
) {
int B = tensor1.size(0);
int C = tensor1.size(1);
int T = tensor1.size(2);
int H = tensor1.size(3);
int W = tensor1.size(4);
const int outer_size = B * C;
const int inner_dim2 = H * W;
const int hw_ind = blockIdx.z * blockDim.x + threadIdx.x;
const int h = hw_ind / W;
const int w = hw_ind % W;
if (h >= H || w >= W) {
return;
}
const int t = blockIdx.y;
const int block_bc = blockIdx.x * BLOCK_ROWS;
const int b = block_bc / C;
const int c = block_bc % C;
for (int i = 0; i < BLOCK_ROWS; i++) {
result[b][c + i][t][h][w] = tensor1[b][c + i][t][h][w] + tensor2[b][c + i][t + offset][h][w];
}
}
void inplace_add_cuda(
at::Tensor& x,
at::Tensor& workspace,
int offset,
cudaStream_t stream
) {
int B = x.size(0);
int C = x.size(1);
int T = x.size(2);
int H = x.size(3);
int W = x.size(4);
int outer_size = B * (C / BLOCK_ROWS);
int inner_dim2 = H * W;
const int max_threads_per_block = 1024;
int threads_per_block_x = (inner_dim2 > max_threads_per_block) ? max_threads_per_block : inner_dim2;
int threads_per_block_y = (T > max_threads_per_block / threads_per_block_x) ? max_threads_per_block / threads_per_block_x : T;
dim3 threadsPerBlock(threads_per_block_x, threads_per_block_y);
int max_v = (threadsPerBlock.x * threadsPerBlock.y);
int inner_dim = ((inner_dim2 + max_v - 1) / max_v);
dim3 blocksPerGrid(outer_size, T, inner_dim);
if (x.numel() >= std::numeric_limits<int>::max() ||
workspace.numel() >= std::numeric_limits<int>::max()) {
add_tensors_tiled<<<blocksPerGrid, threadsPerBlock, 0, stream>>>(
x.packed_accessor64<at::BFloat16, 5, torch::RestrictPtrTraits>(),
workspace.packed_accessor64<at::BFloat16, 5, torch::RestrictPtrTraits>(),
x.packed_accessor64<at::BFloat16, 5, torch::RestrictPtrTraits>(),
offset
);
} else {
add_tensors_tiled<<<blocksPerGrid, threadsPerBlock, 0, stream>>>(
x.packed_accessor32<at::BFloat16, 5, torch::RestrictPtrTraits>(),
workspace.packed_accessor32<at::BFloat16, 5, torch::RestrictPtrTraits>(),
x.packed_accessor32<at::BFloat16, 5, torch::RestrictPtrTraits>(),
offset
);
}
C10_CUDA_KERNEL_LAUNCH_CHECK();
}
void inplace_add_cuda(at::Tensor&, at::Tensor&, int, cudaStream_t);
vae_patcher/kernels/ops.py
-28
View File
@@ -1,28 +0,0 @@
import os
from torch.utils.cpp_extension import load
pixel_norm = load(
name="pixel_norm",
sources=[
os.path.join(os.path.dirname(__file__), "pixel_norm.cpp"),
os.path.join(os.path.dirname(__file__), "pixel_norm_cuda.cu"),
],
)
def pixel_norm_inplace(x, scale, shift, eps=1e-5):
return pixel_norm.pixel_norm_inplace(x, scale, shift, eps)
inplace_add = load(
name="inplace_add",
sources=[
os.path.join(os.path.dirname(__file__), "add_inplace.cpp"),
os.path.join(os.path.dirname(__file__), "add_inplace_cuda.cu"),
],
)
def add_inplace(x, workspace, offset):
return inplace_add.inplace_add(x, workspace, offset)
vae_patcher/kernels/pixel_norm.cpp
-43
View File
@@ -1,43 +0,0 @@
#include <ATen/cuda/CUDAContext.h>
#include <c10/cuda/CUDAGuard.h>
#include <torch/extension.h>
#include <torch/python.h>
#include <vector>
template <typename input_t>
void pixel_norm_cuda(
const void* input,
const void* scale,
const void* shift,
void* output,
const float eps,
const int batch_size,
const int num_channels,
const int num_pixels,
cudaStream_t stream
);
void pixel_norm_inplace(at::Tensor &x, at::Tensor &scale, at::Tensor &shift, float eps) {
int num_pixels = x.size(2)*x.size(3)*x.size(4);
int batch_size = x.size(0);
int num_channels = x.size(1);
at::cuda::CUDAGuard device_guard{(char)x.get_device()};
auto stream = at::cuda::getCurrentCUDAStream().stream();
pixel_norm_cuda<at::BFloat16>(
x.data_ptr(),
scale.data_ptr(),
shift.data_ptr(),
x.data_ptr(),
eps,
batch_size,
num_channels,
num_pixels,
stream
);
}
PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
m.def("pixel_norm_inplace", &pixel_norm_inplace,
"pixel norm inplace");
}
vae_patcher/kernels/pixel_norm_cuda.cu
-161
View File
@@ -1,161 +0,0 @@
#include <c10/util/BFloat16.h>
#include <c10/cuda/CUDAException.h> // For C10_CUDA_CHECK and C10_CUDA_KERNEL_LAUNCH_CHECK
#include <cuda_runtime.h>
#include <cuda_bf16.h>
#include <stdio.h>
template <int CHANNELS_, int NUM_WARPS_, typename input_t_>
struct pixel_norm_kernel_traits {
static constexpr int CHANNELS = CHANNELS_;
static constexpr int NUM_WARPS = NUM_WARPS_;
using input_t = input_t_;
using copy_t = uint4;
static constexpr int NUM_THREADS = NUM_WARPS * 32;
static constexpr int NUM_ELEMS_PER_COPY = sizeof(copy_t) / sizeof(input_t);
static constexpr int NUM_THREADS_PER_PIXEL = CHANNELS / NUM_ELEMS_PER_COPY;
static constexpr int NUM_WARPS_PER_PIXEL = NUM_THREADS_PER_PIXEL / 32;
static constexpr int PIXELS_PER_BLOCK = NUM_THREADS / NUM_THREADS_PER_PIXEL;
};
template <int NUM_ELEMS, typename input_t, typename copy_t, int COPY_THREADS>
inline __device__ void load_input(const input_t *x, float x_vals[NUM_ELEMS]) {;
input_t x_vals_load[NUM_ELEMS] = {0};
reinterpret_cast<copy_t*>(x_vals_load)[0] = reinterpret_cast<const copy_t*>(x)[threadIdx.x % COPY_THREADS];
for (int i = 0; i < NUM_ELEMS; i++) {
x_vals[i] = float(x_vals_load[i]);
}
}
inline __device__ float silu(float x) {
return x / (1.0f + expf(-x));
}
template <typename kernel_traits>
__global__ void pixel_norm_kernel(
const typename kernel_traits::input_t* input,
const typename kernel_traits::input_t* scale,
const typename kernel_traits::input_t* shift,
typename kernel_traits::input_t* output,
const float eps,
const int num_pixels,
const int batch_size
) {
extern __shared__ float smem_[];
const int tid = threadIdx.x;
const int pixel_id = blockIdx.x * kernel_traits::PIXELS_PER_BLOCK;
const int batch_id = pixel_id / num_pixels;
const int total_pixels = num_pixels * batch_size;
const int processing_pixels = pixel_id + tid / kernel_traits::NUM_THREADS_PER_PIXEL;
if (processing_pixels >= total_pixels) {
return;
}
float x_vals[kernel_traits::NUM_ELEMS_PER_COPY];
float sum = 0.0f;
const typename kernel_traits::input_t* x_ptr = input + static_cast<long long>(pixel_id) * static_cast<long long>(kernel_traits::CHANNELS);
typename kernel_traits::input_t* output_ptr = output + static_cast<long long>(pixel_id) * static_cast<long long>(kernel_traits::CHANNELS);
const typename kernel_traits::input_t* scale_ptr = scale + batch_id * kernel_traits::CHANNELS;
const typename kernel_traits::input_t* shift_ptr = shift + batch_id * kernel_traits::CHANNELS;
float scale_vals[kernel_traits::NUM_ELEMS_PER_COPY];
float shift_vals[kernel_traits::NUM_ELEMS_PER_COPY];
load_input<kernel_traits::NUM_ELEMS_PER_COPY, typename kernel_traits::input_t, typename kernel_traits::copy_t, kernel_traits::NUM_THREADS>(x_ptr, x_vals);
load_input<kernel_traits::NUM_ELEMS_PER_COPY, typename kernel_traits::input_t, typename kernel_traits::copy_t, kernel_traits::NUM_THREADS_PER_PIXEL>(scale_ptr, scale_vals);
load_input<kernel_traits::NUM_ELEMS_PER_COPY, typename kernel_traits::input_t, typename kernel_traits::copy_t, kernel_traits::NUM_THREADS_PER_PIXEL>(shift_ptr, shift_vals);
for (int i = 0; i < kernel_traits::NUM_ELEMS_PER_COPY; i++) {
sum += x_vals[i] * x_vals[i];
}
for (int i = 1; i < kernel_traits::NUM_THREADS_PER_PIXEL; i *= 2) {
sum += __shfl_xor_sync(uint32_t(-1), sum, i);
}
if constexpr (kernel_traits::NUM_WARPS_PER_PIXEL > 1) {
smem_[tid / 32] = sum;
__syncthreads();
sum = 0.0f;
for (int i = 0; i < kernel_traits::NUM_WARPS_PER_PIXEL; i++) {
sum += smem_[i];
}
}
sum = sum / kernel_traits::CHANNELS;
float inv_sqrt_sum = rsqrtf(sum + eps);
#pragma unroll
for (int i = 0; i < kernel_traits::NUM_ELEMS_PER_COPY; i++) {
x_vals[i] *= inv_sqrt_sum;
x_vals[i] = (1.0f + scale_vals[i]) * x_vals[i] + shift_vals[i];
x_vals[i] = silu(x_vals[i]);
}
typename kernel_traits::input_t out_vals_store[kernel_traits::NUM_ELEMS_PER_COPY];
for (int i = 0; i < kernel_traits::NUM_ELEMS_PER_COPY; i++) {
out_vals_store[i] = static_cast<typename kernel_traits::input_t>(x_vals[i]);
}
reinterpret_cast<typename kernel_traits::copy_t*>(output_ptr)[tid] = reinterpret_cast<const typename kernel_traits::copy_t*>(out_vals_store)[0];
}
template <int NUM_WARPS, int CHANNELS, typename input_t, typename output_t>
void pixel_norm_launch(
const void* input,
const void* scale,
const void* shift,
void* output,
const float eps,
const int batch_size,
const int num_pixels,
cudaStream_t stream
) {
using kernel_traits = pixel_norm_kernel_traits<CHANNELS, NUM_WARPS, input_t>;
int BLOCK_SIZE = (num_pixels*batch_size + kernel_traits::PIXELS_PER_BLOCK - 1) / kernel_traits::PIXELS_PER_BLOCK;
dim3 block(kernel_traits::NUM_THREADS);
dim3 grid(BLOCK_SIZE);
auto kernel = &pixel_norm_kernel<kernel_traits>;
kernel<<<grid, block, 8192, stream>>>(
(const typename kernel_traits::input_t*)input,
(const typename kernel_traits::input_t*)scale,
(const typename kernel_traits::input_t*)shift,
(typename kernel_traits::input_t*)output,
eps,
num_pixels,
batch_size
);
C10_CUDA_KERNEL_LAUNCH_CHECK();
}
template <typename input_t>
void pixel_norm_cuda(
const void* input,
const void* scale,
const void* shift,
void* output,
const float eps,
const int batch_size,
const int num_channels,
const int num_pixels,
cudaStream_t stream
) {
if (num_channels == 128) {
pixel_norm_launch<2, 128, input_t, input_t>(input, scale, shift, output, eps, batch_size, num_pixels, stream);
} else if (num_channels == 256) {
pixel_norm_launch<4, 256, input_t, input_t>(input, scale, shift, output, eps, batch_size, num_pixels, stream);
} else if (num_channels == 512) {
pixel_norm_launch<4, 512, input_t, input_t>(input, scale, shift, output, eps, batch_size, num_pixels, stream);
} else if (num_channels == 1024) {
pixel_norm_launch<8, 1024, input_t, input_t>(input, scale, shift, output, eps, batch_size, num_pixels, stream);
}
}
template void pixel_norm_cuda<at::BFloat16>(const void*, const void*, const void*, void*, float, int, int, int, cudaStream_t);
vae_patcher/patch.py
-35
View File
@@ -1,35 +0,0 @@
import types
import torch
from .forwards import (
block_forward,
decoder_forward,
res_block_forward,
upsample_forward,
)
def patch_vae(vae_model, patch_block=4):
vae_model.decoder.forward = types.MethodType(decoder_forward, vae_model.decoder)
vae_model.decoder.conv_out.conv.weight.data = (
vae_model.decoder.conv_out.conv.weight.data.to(
memory_format=torch.channels_last_3d
)
)
for i, block in enumerate(reversed(vae_model.decoder.up_blocks)):
if i >= patch_block:
break
if block.__class__.__name__ == "UNetMidBlock3D":
block.forward = types.MethodType(block_forward, block)
for name, param in block.named_parameters():
if "conv1" in name or "conv2" in name:
if "weight" in name:
param = param.to(memory_format=torch.channels_last_3d)
for res_block in block.res_blocks:
if res_block.__class__.__name__ == "ResnetBlock3D":
res_block.forward = types.MethodType(res_block_forward, res_block)
elif block.__class__.__name__ == "DepthToSpaceUpsample":
block.forward = types.MethodType(upsample_forward, block)
vae_patcher/vae_patcher.py
-38
View File
@@ -1,38 +0,0 @@
from comfy.model_management import dtype_size
from ..nodes_registry import comfy_node
@comfy_node(name="LTXVPatcherVAE")
class LTXVPatcherVAE:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"vae": ("VAE",),
}
}
RETURN_TYPES = ("VAE",)
FUNCTION = "patch"
CATEGORY = "lightricks/LTXV"
TITLE = "LTXV VAE Patcher"
def patch(self, vae):
from .patch import patch_vae
vae_model = vae.first_stage_model
vae.memory_used_decode = (
lambda shape, dtype: shape[1]
* shape[2]
* shape[3]
* shape[4]
* 8
* 8
* 8
* 3
* dtype_size(dtype)
)
patch_vae(vae_model, patch_block=4)
return (vae,)