CavinHuang-comfyui-nodes-docs/nodes-docs/en-US/SaltAI_Language_Toolkit/docs/LLMSimpleWebPageReader.md

∞ Simple Web Page

Documentation

• Class name: LLMSimpleWebPageReader
• Category: SALT/Language Toolkit/Readers
• Output node: False

The LLMSimpleWebPageReader node is designed to fetch and process web page content from specified URLs, converting HTML content to text if desired. It allows for the extraction of web page data by providing up to four URLs, making it a versatile tool for web content retrieval and preprocessing.

Input types

Required

• url_i
  • unknown
  • Comfy dtype: STRING
  • Python dtype: unknown

Optional

• html_to_text
  • A boolean flag indicating whether to convert HTML content to plain text, facilitating easier processing and analysis of the web page data.
  • Comfy dtype: BOOLEAN
  • Python dtype: bool

Output types

• documents
  • Comfy dtype: DOCUMENT
  • The processed web page content, returned as documents. This output provides the extracted data from the specified URLs, potentially converted to text.
  • Python dtype: tuple

Usage tips

• Infra type: CPU
• Common nodes: unknown

Source code

class LLMSimpleWebPageReader:
    @classmethod
    def INPUT_TYPES(cls):
        return {
            "required": {
                "url_1": ("STRING", {}),
            },
            "optional": {
                "url_2": ("STRING", {}),
                "url_3": ("STRING", {}),
                "url_4": ("STRING", {}),
                "html_to_text": ("BOOLEAN", {"default": True}),
            },
        }

    RETURN_TYPES = ("DOCUMENT", )
    RETURN_NAMES = ("documents",)

    FUNCTION = "read_web"
    CATEGORY = f"{MENU_NAME}/{SUB_MENU_NAME}/Readers"

    def read_web(self, url_1, url_2=None, url_3=None, url_4=None, html_to_text=True):
        if not url_1.strip():
            raise ValueError("At least one URL must be provided to LLMSimpleWebPageReader")

        urls = [url_1.strip()]
        if url_2.strip():
            urls.append(url_2.strip())
        if url_3.strip():
            urls.append(url_3.strip())
        if url_4.strip():
            urls.append(url_4.strip())

        valid_urls = []
        for url in urls:
            if not valid_url(url):
                print("Skipping invalid URL", url)
                continue
            valid_urls.append(url)

        print("Valided URLs:", valid_urls)

        documents = SimpleWebPageReader(html_to_text=html_to_text).load_data(valid_urls)
        return (documents,)