72 lines
2.0 KiB
Markdown
72 lines
2.0 KiB
Markdown
|
|
# Documentation
|
|
- Class name: LLMSimpleWebPageReaderAdv
|
|
- Category: SALT/Language Toolkit/Readers
|
|
- Output node: False
|
|
- Repo Ref: https://github.com/comfyanonymous/ComfyUI
|
|
|
|
LLMSimpleWebPageReaderAdv节点用于从URL列表中获取并处理网页，将其转换为结构化文档格式。它可以选择将HTML内容转换为纯文本，便于从网页中提取可读内容以供进一步分析或处理。
|
|
|
|
# Input types
|
|
## Required
|
|
- urls
|
|
- 一个URL列表，用于获取网页内容。这是驱动节点操作的主要输入，决定了要处理的网页。
|
|
- Comfy dtype: LIST
|
|
- Python dtype: List[str]
|
|
## Optional
|
|
- html_to_text
|
|
- 一个布尔标志，指示是否将HTML内容转换为纯文本。这会影响网页的处理过程，去除HTML标签，只保留可读文本。
|
|
- Comfy dtype: BOOLEAN
|
|
- Python dtype: bool
|
|
|
|
# Output types
|
|
- documents
|
|
- 结构化文档格式的输出，包含从获取的网页中处理得到的内容。
|
|
- Comfy dtype: DOCUMENT
|
|
- Python dtype: Tuple[Document]
|
|
|
|
|
|
## Usage tips
|
|
- Infra type: `CPU`
|
|
- Common nodes: unknown
|
|
|
|
|
|
## Source code
|
|
```python
|
|
class LLMSimpleWebPageReaderAdv:
|
|
@classmethod
|
|
def INPUT_TYPES(cls):
|
|
return {
|
|
"required": {
|
|
"urls": ("LIST", {}),
|
|
},
|
|
"optional": {
|
|
"html_to_text": ("BOOLEAN", {"default": True}),
|
|
},
|
|
}
|
|
|
|
RETURN_TYPES = ("DOCUMENT", )
|
|
RETURN_NAMES = ("documents",)
|
|
|
|
FUNCTION = "read_web"
|
|
CATEGORY = f"{MENU_NAME}/{SUB_MENU_NAME}/Readers"
|
|
|
|
def read_web(self, urls, html_to_text=True):
|
|
|
|
if not urls:
|
|
raise ValueError("At least one URL must be provided to LLMSimpleWebPageReaderAdv")
|
|
|
|
valid_urls = []
|
|
for url in urls:
|
|
if not valid_url(url):
|
|
print("Skipping invalid URL", url)
|
|
continue
|
|
valid_urls.append(url)
|
|
|
|
print("Valided URLs:", valid_urls)
|
|
|
|
documents = SimpleWebPageReader(html_to_text=html_to_text).load_data(valid_urls)
|
|
return (documents,)
|
|
|
|
```
|