109 lines
5.5 KiB
Markdown
109 lines
5.5 KiB
Markdown
I've reviewed the code and will help you understand how these components work together and verify the integration.
|
|
|
|
## Integration Check
|
|
|
|
Looking at the code, there are a few issues that need to be fixed to ensure the `GenericWebsiteWithAuthHandler` is properly used:
|
|
|
|
1. In `web-image-scraper-node_v7.py`, it looks like you've modified the `_get_handler_for_url` method to call `GenericWebsiteWithAuthHandler` as a fallback, but this needs to be fixed:
|
|
|
|
```python
|
|
def _get_handler_for_url(self, url):
|
|
"""Get the appropriate handler for the given URL"""
|
|
for HandlerClass in self.site_handlers.values():
|
|
if HandlerClass.can_handle(url):
|
|
handler_instance = HandlerClass(url, self)
|
|
return handler_instance
|
|
# Default fallback if no handler is found
|
|
return GenericWebsiteWithAuthHandler(url, self)
|
|
```
|
|
|
|
However, this assumes `GenericWebsiteWithAuthHandler` is in scope, but it isn't imported directly in this file. You would need to ensure the class is either:
|
|
|
|
1. Properly registered in the `loaded_handlers` dictionary (which is what you're currently trying to do), or
|
|
2. Directly imported at the top of the file
|
|
|
|
Since you're trying to use the first approach, you need to make sure the handler is correctly loaded via the `load_site_handlers()` function. Your handler should be placed in the `site_handlers` directory so it's found during this scan.
|
|
|
|
## How These Components Work Together
|
|
|
|
Let me explain how the main node, base handler, and generic handler with auth work together:
|
|
|
|
### 1. Architecture Overview
|
|
|
|
- **Web Image Scraper Node** (`web-image-scraper-node_v7.py`): This is the main ComfyUI node that provides the UI interface and orchestrates the scraping process.
|
|
|
|
- **Base Handler** (`base_handler.py`, not shown): Defines the abstract base class that all site handlers must inherit from. It defines the interface for handlers with methods like `can_handle`, `extract_with_direct_playwright`, etc.
|
|
|
|
- **GenericWebsiteWithAuthHandler** (`generic_handler_with_auth.py`): A specialized handler that inherits from `BaseSiteHandler` and provides implementations for handling authentication and generic website content extraction.
|
|
|
|
### 2. Flow of Operation
|
|
|
|
1. **Component Initialization**:
|
|
- The Web Scraper Node initializes and loads all available site handlers using `load_site_handlers()`
|
|
- Handlers are registered in the `loaded_handlers` dictionary
|
|
- The `SessionManager` is initialized for managing authentication sessions
|
|
|
|
2. **Scraper Execution**:
|
|
- When a URL is provided, `_get_handler_for_url()` is called to find an appropriate handler
|
|
- Each handler's `can_handle(url)` method is checked in order of priority
|
|
- Site-specific handlers (Reddit, Bsky, etc.) are checked first
|
|
- `GenericWebsiteWithAuthHandler` is used as a fallback since its `can_handle()` returns `True` for all URLs
|
|
|
|
3. **Authentication Process**:
|
|
- When a handler is selected, it receives a reference to the scraper (`self`) which allows it to access:
|
|
- The authentication configuration
|
|
- The session manager
|
|
- Other global settings
|
|
|
|
- During extraction with Playwright, the handler can:
|
|
- Check if there's a saved session for the domain
|
|
- Try to load the session
|
|
- If no valid session exists, perform login using credentials from auth_config
|
|
- Save the new session for future use
|
|
|
|
4. **Content Extraction**:
|
|
- After authentication (if needed), the handler extracts media using multiple methods:
|
|
- Browser cache extraction
|
|
- Network resource monitoring
|
|
- DOM traversal
|
|
- HTML parsing
|
|
- Card gallery detection and extraction
|
|
|
|
5. **Integration Points**:
|
|
- The authentication code in `GenericWebsiteWithAuthHandler` is called from:
|
|
- `pre_process_async`: Initial setup and login
|
|
- `extract_with_direct_playwright_async`: Before extraction begins
|
|
- The main `_async_scrape_files` method's extraction strategies
|
|
|
|
### 3. Authentication How-To
|
|
|
|
When a site requires authentication:
|
|
|
|
1. Add credentials to `auth_config.json` for the domain
|
|
2. The scraper checks if a domain has auth configuration
|
|
3. `GenericWebsiteWithAuthHandler` is selected for generic sites
|
|
4. It attempts to load any existing session first
|
|
5. If no session exists, it performs login through various means
|
|
6. After successful login, it saves the session cookies
|
|
7. Future visits to the same domain will reuse the authenticated session
|
|
|
|
## Specific Integration Issues
|
|
|
|
Looking at your current integration, here are the issues to fix:
|
|
|
|
1. **Class Name Mismatch**: In the main node's fallback, you refer to `GenericWebsiteWithAuthHandler` but your class is actually named `GenericWebsiteWithAuthHandler`. These need to match.
|
|
|
|
2. **Parameter Handling**: The `generic_items` parameter you noticed is used for fallback extraction when a site-specific handler doesn't find enough items. This is separate from the handler selection.
|
|
|
|
3. **Handler Registration**: Make sure your handler is properly registered via the site handler loading mechanism, which means it needs to be in the site_handlers directory.
|
|
|
|
## Recommendation
|
|
|
|
To properly integrate your new handler:
|
|
|
|
1. Place `generic_handler_with_auth.py` in the `site_handlers` directory
|
|
2. Make sure it's named correctly and extends `BaseSiteHandler`
|
|
3. Make sure its `can_handle` method returns `True` but with a lower priority than specialized handlers
|
|
4. Verify that the `GenericWebsiteWithAuthHandler` class is being found in the loader's output
|
|
|
|
With these changes, your authentication-enabled generic handler should be properly used as a fallback when no other handler can process a URL. |