74 lines
4.2 KiB
Markdown
74 lines
4.2 KiB
Markdown
# Graph200K Dataset
|
|
|
|
1. [Introduction](#introduction)
|
|
2. [Usage](#usage)
|
|
3. [Preprocessing](#preprocessing)
|
|
|
|
## Introduction
|
|
|
|
In natural language processing, tasks overlap significantly,
|
|
facilitating strong cross-task learning ability. In contrast,
|
|
visual tasks are inherently distinct, posing challenges for
|
|
vision models to achieve similar generalization ability via
|
|
instruction tuning. To ease this issue, we introduce a Graph
|
|
Structured Multi-Task Dataset, named Graph200K.
|
|
|
|
Graph200K is built upon the [Subjects200K](https://huggingface.co/datasets/Yuanshi/Subjects200K) dataset.
|
|
Each image is annotated for five meta-tasks, including
|
|
**1) conditional generation, 2) image restoration, 3) image editing, 4) IP preservation, and 5) style transfer**.
|
|
Using these tasks, we can also combine a wide range of complex
|
|
tasks, as shown in the figure below.
|
|
|
|
|
|
|
|
## Usage
|
|
|
|
The dataset can be downloaded and used through the [Datasets](https://huggingface.co/docs/datasets/index) library, as follows:
|
|
|
|
```python
|
|
import datasets
|
|
grapth200k = datasets.load_dataset("lzyhha/test") # todo
|
|
|
|
train = grapth200k['train']
|
|
test = grapth200k['test']
|
|
|
|
# Reading depth map (PIL.Image) of the first image in the train set
|
|
train[0]['depth'].save(f'depth.jpg')
|
|
```
|
|
|
|
In each item of the dataset, there are annotations as follows. Examples can be found in [huggingface](https://huggingface.co/datasets/VisualCloze/Graph200K).
|
|
|
|
| Item | Meaning |
|
|
| :------------------------ | ------------------------------------------------------------- |
|
|
| ref | Inherited from Subjects200K, it depicts the subject object in the target image. |
|
|
| target | The original image inherited from Subjects200K. |
|
|
| InstantStyle_image_{0-3} | Stylized images with invariant semantics. |
|
|
| InstantStyle_ref_{0-3} | Style reference for InstantStyle. |
|
|
| ReduxStyle_image_{0-3} | Stylized images with variant semantics. |
|
|
| ReduxStyle_ref_{0-3} | Style reference for ReduxStyle. |
|
|
| FillEdit_image_{0-5} | Edited image with invariant background. |
|
|
| FillEdit_meta | The name and descripation of the new subject object after editing. |
|
|
| DepthEdit | Edited image with variant background. |
|
|
| qwen_2_5_mask | A high-quality segmentation mask generated by the [Qwen-2.5-VL](https://github.com/QwenLM/Qwen2.5-VL) and [SAM2](https://github.com/facebookresearch/sam2). |
|
|
| qwen_2_5_bounding_box | The bounding boxes generated by the [Qwen-2.5-VL](https://github.com/QwenLM/Qwen2.5-VL). |
|
|
| qwen_2_5_meta | The coordinate and object name of each bounding box. And the mask color corresponding ro each box. |
|
|
| sam2_mask | A mask generated by the [SAM2](https://github.com/facebookresearch/sam2) model. |
|
|
| uniformer | The semantic segmentation generated by [UniFormer](https://github.com/Sense-X/UniFormer). |
|
|
| foreground | The foreground mask generated by [RMBG-2.0](https://huggingface.co/briaai/RMBG-2.0). |
|
|
| normal | Surface normal estimation generated by [DSINE](https://github.com/baegwangbin/DSINE/tree/main) |
|
|
| depth | The depth estimation by [Depth Anything V2](https://github.com/DepthAnything/Depth-Anything-V2). |
|
|
| canny | Edge detection in images, using the Canny edge detector. |
|
|
| hed |Edge detection in images, using the [HED](https://github.com/s9xie/hed) detector. |
|
|
| mlsd | Line segments generated using [M-LSD](https://github.com/navervision/mlsd). |
|
|
| openpose | Human keypoints generated by [OpenPose](https://github.com/CMU-Perceptual-Computing-Lab/openpose) |
|
|
|
|
## Preprocessing
|
|
|
|
To use the Graph200k for training and inference in our VisualCloze, we extract each image fileand generate a json file that record the path and meta information of each file.
|
|
|
|
```bash
|
|
python processing.py \
|
|
--target_path "where will the images and the json be saved" \
|
|
--split "train or test"
|
|
```
|