136 lines
5.5 KiB
Markdown
136 lines
5.5 KiB
Markdown
|
|
# Documentation
|
|
- Class name: InstanceDiffusionTrackingPrompt
|
|
- Category: instance/conditioning
|
|
- Output node: False
|
|
|
|
InstanceDiffusionTrackingPrompt节点旨在将基于文本的提示与实例跟踪信息相结合，以便为实例扩散模型生成条件输入。它提取并编码与特定实例或类别相关的提示，将这些提示与空间跟踪数据结合，并为实例扩散机制的进一步处理做好准备。
|
|
|
|
# Input types
|
|
## Required
|
|
- positive
|
|
- 'positive'参数代表正面实例的初始条件数据，作为应用基于文本条件的基础。
|
|
- Comfy dtype: CONDITIONING
|
|
- Python dtype: dict
|
|
- negative
|
|
- 与'positive'类似，'negative'参数保存负面实例的初始条件数据，这些数据将被基于文本的条件增强。
|
|
- Comfy dtype: CONDITIONING
|
|
- Python dtype: dict
|
|
- clip
|
|
- 'clip'参数是一个用于将文本提示编码为嵌入的模型，在解释文本信息方面起着至关重要的作用。
|
|
- Comfy dtype: CLIP
|
|
- Python dtype: object
|
|
- tracking
|
|
- 此参数包含实例的空间跟踪信息，对于将文本提示与其相应的空间位置对齐至关重要。
|
|
- Comfy dtype: TRACKING
|
|
- Python dtype: dict
|
|
- positionnet
|
|
- PositionNet是一个模型，与fusers结合使用，用于细化实例的空间信息以获得更好的条件设置。
|
|
- Comfy dtype: POSITIONNET
|
|
- Python dtype: object
|
|
- fusers
|
|
- Fusers是整合多个信息源（如嵌入和空间数据）的机制，用于增强条件设置过程。
|
|
- Comfy dtype: FUSERS
|
|
- Python dtype: object
|
|
- positive_text
|
|
- 一个多行字符串，包含与正面实例相关的文本提示，这些提示将被处理和编码。
|
|
- Comfy dtype: STRING
|
|
- Python dtype: str
|
|
- negative_text
|
|
- 一个多行字符串，包含与负面实例相关的文本提示，将以与positive_text类似的方式进行处理和编码。
|
|
- Comfy dtype: STRING
|
|
- Python dtype: str
|
|
|
|
# Output types
|
|
- positive
|
|
- 输出是正面实例的条件数据，现在已经融合了基于文本的提示和空间跟踪信息，可以在实例扩散模型中进行进一步处理。
|
|
- Comfy dtype: CONDITIONING
|
|
- Python dtype: list
|
|
- negative
|
|
- 类似地，这个输出是负面实例的条件数据，也融合了基于文本的提示和空间跟踪信息。
|
|
- Comfy dtype: CONDITIONING
|
|
- Python dtype: list
|
|
|
|
|
|
## Usage tips
|
|
- Infra type: `CPU`
|
|
- Common nodes: unknown
|
|
|
|
|
|
## Source code
|
|
```python
|
|
class InstanceDiffusionTrackingPromptNode:
|
|
@classmethod
|
|
def INPUT_TYPES(s):
|
|
return {"required": {"positive": ("CONDITIONING", ),
|
|
"negative": ("CONDITIONING", ),
|
|
"clip": ("CLIP", ),
|
|
"tracking": ("TRACKING", ),
|
|
"positionnet": ("POSITIONNET", ),
|
|
"fusers": ("FUSERS", ),
|
|
"positive_text": ("STRING", {"multiline": True}),
|
|
"negative_text": ("STRING", {"multiline": True}),
|
|
}}
|
|
RETURN_TYPES = ("CONDITIONING", "CONDITIONING")
|
|
RETURN_NAMES = ("positive", "negative")
|
|
FUNCTION = "append"
|
|
|
|
CATEGORY = "instance/conditioning"
|
|
|
|
def _get_position_conds(self, clip, tracking, text):
|
|
# Get prompts and their class id and trakcer id
|
|
prompt_pairs = extract_prompts(text)
|
|
|
|
# Go through prompt pairs, encode prompts, and join with positions from tracking
|
|
position_conds = []
|
|
for tracker_id, class_id, prompt in prompt_pairs:
|
|
_, cond_pooled = clip.encode_from_tokens(
|
|
clip.tokenize(prompt), return_pooled=True)
|
|
# A tracker_id of -1 means that it is prompting all instances of a single class
|
|
if tracker_id != -1:
|
|
position_cond = {'cond_pooled': cond_pooled, 'positions':
|
|
tracking[class_id][tracker_id]}
|
|
position_conds.append(position_cond)
|
|
else:
|
|
for tracker_id in tracking[class_id]:
|
|
position_cond = {'cond_pooled': cond_pooled,
|
|
'positions': tracking[class_id][tracker_id]}
|
|
position_conds.append(position_cond)
|
|
|
|
return position_conds
|
|
|
|
def _apply_position_conds(self, position_conds, conditioning, fusers, positionnet):
|
|
# Add prompts+embeddings to the input conditionings
|
|
cond_out = []
|
|
for t in conditioning:
|
|
n = [t[0], t[1].copy()]
|
|
cond = n[1]
|
|
prev = []
|
|
has_instance = 'instance_diffusion' in cond
|
|
instance_conditioning = conditioning['instance_diffusion'] if has_instance else InstanceConditioning(
|
|
fusers, positionnet)
|
|
cond['instance_diffusion'] = instance_conditioning
|
|
instance_conditioning.add_conds(position_conds)
|
|
|
|
cond['gligen'] = ('position', instance_conditioning, None)
|
|
|
|
cond_out.append(n)
|
|
|
|
return cond_out
|
|
|
|
def append(self, positive, negative, clip, tracking, fusers, positionnet, positive_text, negative_text, fusers_batch_size=None):
|
|
|
|
positive_positions = self._get_position_conds(
|
|
clip, tracking, positive_text)
|
|
positive = self._apply_position_conds(
|
|
positive_positions, positive, fusers, positionnet)
|
|
|
|
negative_positions = self._get_position_conds(
|
|
clip, tracking, negative_text)
|
|
negative = self._apply_position_conds(
|
|
negative_positions, negative, fusers, positionnet)
|
|
|
|
return (positive, negative)
|
|
|
|
```
|