From 299bec5ccb09c40577618cabd2d5033c235fed3c Mon Sep 17 00:00:00 2001 From: Fred Bliss Date: Mon, 27 Jan 2025 12:58:29 -0600 Subject: [PATCH] reorg --- README.md | 106 ++++++++++++++++------ data_utils/README.md | 82 +++++++++++++++++ data_utils/{ => mira}/process_miradata.py | 0 3 files changed, 161 insertions(+), 27 deletions(-) create mode 100644 data_utils/README.md rename data_utils/{ => mira}/process_miradata.py (100%) diff --git a/README.md b/README.md index 862939c..290466d 100644 --- a/README.md +++ b/README.md @@ -1,58 +1,110 @@ -# ComfyUI Dataset Batch Node +# ComfyUI Dataset Helper & Batch Node This custom node set for ComfyUI provides a `DatasetBatchNode` for automated, sequential processing of datasets, particularly useful for iterative training or batched image/video generation workflows. +## Usage + **Key Features:** -* Loads datasets from local files (JSONL, CSV) or Hugging Face `datasets`. -* Processes individual rows sequentially, triggering a new job for each. -* `magic_number` input automatically tracks the current row index across multiple jobs, ensuring no rows are missed or duplicated, even if you stop and restart the workflow. -* Supports a flexible `mixed_fields_config_json` input to define complex prompt construction from multiple dataset fields and filters. -* Includes a script (`process_miradata.py`) to prepare datasets for use with the node. +* Loads datasets from local files (JSONL, CSV) or Hugging Face `datasets`. +* Processes individual rows sequentially, triggering a new job for each. +* `magic_number` input automatically tracks the row index across multiple jobs, ensuring seamless dataset iteration even across workflow restarts. +* Offers flexible prompt construction using `mixed_fields_config_json` to combine dataset fields and apply filters. +* *data utilities*: currently only includes `process_miradata.py` script to preprocess the MiraData dataset for this node, but really can use any dataset. These can be found in `./data_utils/` with a separate `README.md` on how to use that. **Node Configuration (`DatasetBatchNode`)** -* **`dataset_path`:** (Required) Path to your dataset file (JSONL, CSV) or the Hugging Face dataset name. -* **`prompt_field`:** (Required) The field in your dataset containing the base prompt text. -* **`num_rows`:** (Required) The number of rows to process (-1 for all rows). The node will stop after processing this number of rows -* **`start_row`:** (Required) The starting row index (0-based). -* **`random_seed`:** (Required) Seed for shuffling. -* **`shuffle`:** (Required) Whether to shuffle the rows before processing. -* **`magic_number`:** (Optional) An automatically managed offset that ensures sequential processing across multiple workflow runs. **Do not modify this manually.** Let the node and extension manage it. -* **`delimiter`:** (Optional) Delimiter to use when combining multiple prompt fields (default: `\n`). -* **`text_input`:** (Optional) Text to append before the prompt. -* **`mixed_fields_config_json`:** (Optional) A JSON string defining how to combine multiple fields from your dataset into a single prompt. See the "Advanced Prompt Construction" section below. +* **`dataset_path`:** (Required) Path to your dataset file (JSONL, CSV) or a Hugging Face dataset identifier (e.g., `TencentARC/MiraData`). +* **`prompt_field`:** (Required) Name of the dataset field containing the base text prompt (e.g., `combined_caption`). +* **`num_rows`:** (Required) Total rows to process. Use `-1` to process the entire dataset. +* **`start_row`:** (Required) Row index to begin processing from (0-indexed). +* **`random_seed`:** (Required) Seed for dataset shuffling, if enabled. +* **`shuffle`:** (Required) Enable dataset shuffling before processing. +* **`magic_number`:** (Optional, but **connect a Constant node with value 0**). This input is automatically managed; connect a Constant Number node set to `0` and leave it connected (although you can probably just leave it disconnected too - need to test more). +* **`delimiter`:** (Optional) Separator used to join combined caption fields (default: newline `\n`). +* **`text_input`:** (Optional) Prepended text added to every generated prompt. +* **`mixed_fields_config_json`:** (Optional) JSON configuration for advanced prompt construction. See "Advanced Prompt Construction" below. -**Important:** The `magic_number` input is used for processing each row. There might be a smarter way to do this in Comfy but that's where I landed for now. The JS extension (`dataset_batch_automation.js`) handles the automatic incrementing of this value to ensure proper batch progression. +**Important:** Connect an `INT Constant` node set to `0` to the `magic_number` input. This node is managed automatically by the extension and is crucial for sequential processing. **Advanced Prompt Construction (`mixed_fields_config_json`)** -This optional input allows you to create complex prompts by combining multiple fields from your dataset, potentially applying filters to each field. +Use this optional JSON input for sophisticated prompt creation by combining and filtering different fields from your dataset. -**Format:** +**JSON Configuration Format:** ```json [ { "field": "field_name_1", - "filter": "filter_expression_1" + "filter": "filter_expression_1" // Optional filter }, { "field": "field_name_2", - "filter": "filter_expression_2" - } + "filter": "filter_expression_2" // Optional filter + }, + // etc etc ] +``` +* **`field` (Required):** The name of a field in your dataset (e.g., `"dense_caption"`, `"short_caption"`). +* **`filter` (Optional):** A Python expression string used to conditionally include data from this field. The expression is evaluated for each row, with `example` representing the current dataset row (dictionary). If the expression evaluates to `True`, the field's value is included in the combined prompt. + +## JSON Examples:** + +Assuming you are using the "TencentARC/MiraData" dataset, here are some practical examples: + +### Example 1: Combine `dense_caption` and `style_caption`** + +```json [ { - "field": "description", - "filter": "len(example['description']) > 20" + "field": "dense_caption", + "filter": "" }, { - "field": "tags", - "filter": "'cat' in example['tags']" + "field": "style_caption", + "filter": "" } -]``` +] +``` +This configuration will create prompts by concatenating the `dense_caption` and `style_caption` fields from each row, separated by the `delimiter` you specify in the node (default is newline). +### Example 2: Use `short_caption` for clip IDs >= 7.0, otherwise use `camera_caption`** +```json +[ + { + "field": "short_caption", + "filter": "example['clip_id'] >= 7.0" + }, + { + "field": "camera_caption", + "filter": "example['clip_id'] < 7.0" + } +] +``` + +This example demonstrates conditional prompt creation. For dataset rows where `clip_id` is 7.0 or greater, the `short_caption` will be used. For rows with `clip_id` less than 7.0, the `camera_caption` will be used instead. + +### Example 3: Combine all caption fields** + +```json +[ + { "field": "dense_caption" }, + { "field": "main_object_caption" }, + { "field": "background_caption" }, + { "field": "camera_caption" }, + { "field": "style_caption" } +] +``` + +```markdown +This configuration combines all five available caption fields from the MiraData dataset into a single, detailed prompt. + +**Important Notes:** + +* The `DatasetBatchNode` is designed to work in conjunction with the provided `dataset_batch_automation.js` JavaScript extension. Ensure both the Python node and the JavaScript extension are installed correctly in your `ComfyUI/custom_nodes` directory. +* The `magic_number` input of the `DatasetBatchNode` should be connected to a Constant Number node, but should **not** be manually modified. The automation script manages this value. +``` diff --git a/data_utils/README.md b/data_utils/README.md new file mode 100644 index 0000000..97ff946 --- /dev/null +++ b/data_utils/README.md @@ -0,0 +1,82 @@ +# Data Utilities (`./data_utils/`) + +This directory contains utility scripts for working with various datasets used in ComfyUI workflows. Each subfolder typically represents a specific dataset and contains scripts to download, process, and prepare the data for use in ComfyUI nodes. + +## Currently Supported Datasets + +**MiraData:** Tools for processing the MiraData dataset from TencentARC. + +## General Usage + +The scripts in each subfolder are typically designed to be run from the command line and may take various arguments to customize the processing. You can usually run them with the `--help` flag to see available options: + +```bash +python process_miradata.py --help +``` + +## Dataset-Specific Instructions + +### MiraData + +The `process_miradata.py` script is used to process the MiraData dataset. It allows you to: + +1. **Combine caption fields:** Combine multiple caption fields (e.g., `dense_caption`, `main_object_caption`, etc.) into a single `combined_caption` field. You can specify which fields to combine and the order of combination. +2. **Filter and select rows:** Process all rows, a specific number of rows, a range of rows, or randomly selected rows. +3. **Save in different formats:** Save the processed dataset as a CSV, a Hugging Face dataset, or a JSONL file. +4. **Chunk large datasets:** Process and save large datasets in chunks for efficient memory management (currently only for CSV and JSONL output). +5. **Output only combined captions** Save a file that has only the combined caption field. + +**Usage Example:** + +To combine the `dense_caption`, `main_object_caption`, and `style_caption` fields from the MiraData dataset and save the first 1000 rows to a Hugging Face dataset, you would run: + +```bash +python process_miradata.py --output_type hf --output_folder ./mira_data_processed --output_name MiraData_Combined --num_rows 1000 --fields_to_combine dense_caption main_object_caption style_caption +``` + +```markdown +**Available Arguments:** + +usage: process_miradata.py [-h] [--output_type {csv,hf,jsonl}] [--output_folder OUTPUT_FOLDER] + [--output_name OUTPUT_NAME] [--fields_to_combine FIELDS_TO_COMBINE [FIELDS_TO_COMBINE ...]] + [--num_rows NUM_ROWS] [--row_range ROW_RANGE ROW_RANGE] [--random_rows] + [--chunk_size CHUNK_SIZE] [--only_combined_caption] [--no_header] + +Combine captions in the MiraData dataset and save the result. + +optional arguments: + -h, --help show this help message and exit + --output_type {csv,hf,jsonl} + Output format: 'csv', 'hf' (Hugging Face dataset), or 'jsonl' (default: hf) + --output_folder OUTPUT_FOLDER + Output folder (optional). If not specified, files are saved in the current directory. + --output_name OUTPUT_NAME + Base output name for the resulting file(s) or directory (default: MiraData_combined) + --fields_to_combine FIELDS_TO_COMBINE [FIELDS_TO_COMBINE ...] + List of fields to combine, separated by spaces. Order determines combination order. (default: + ['dense_caption', 'main_object_caption', 'background_caption', 'camera_caption', + 'style_caption']) + --num_rows NUM_ROWS Number of rows to process (default: all rows) + --row_range ROW_RANGE ROW_RANGE + Range of rows to process (e.g., --row_range 10 100 to process rows 10 to 99). + --random_rows Select rows randomly (to be used with --num_rows) + --chunk_size CHUNK_SIZE + Chunk size for processing large datasets (optional) + --only_combined_caption + Output only the combined_caption field + --no_header Do not include a header row in the CSV output (only for CSV output) +``` + +**Primary Dependencies:** + +* `datasets` (Hugging Face Datasets library) +* `pandas` +* `torch` (required by `datasets`) + +**General Guidelines for New Dataset Scripts:** + +* Use command-line arguments (e.g., with `argparse`) to allow users to customize the processing. +* Provide clear instructions and examples in the script's docstrings and in this README. +* Consider supporting common output formats like CSV, JSONL, and Hugging Face Datasets. +* Implement chunking or other memory management techniques for large datasets. +* Follow best practices for data processing and error handling. diff --git a/data_utils/process_miradata.py b/data_utils/mira/process_miradata.py similarity index 100% rename from data_utils/process_miradata.py rename to data_utils/mira/process_miradata.py