lee09lee26-ComfyUI-AsymQuantSampler/eval

Evaluation Toolkit

This directory contains all scripts needed to reproduce the experiments in our paper. The pipeline has three stages: generate images, measure metrics, and plot figures.

Prerequisites

pip install -r requirements.txt

A running ComfyUI server is required for all generation scripts. Measurement and plotting scripts run standalone.

# ComfyUI must be accessible at localhost:8188
# Required custom nodes: ComfyUI-GGUF, AsymQuantSampler (this repo)

Required Model Files

Place these in your ComfyUI models/ directories:

File	Location
flux1-dev-Q2_K.gguf ~ flux1-dev-Q8_0.gguf	models/diffusion_models/
t5-v1_1-xxl-encoder-Q8_0.gguf	models/clip/
clip_l.safetensors	models/clip/
ae.safetensors	models/vae/

Quick Start (Full Reproduction)

# 1. Generate uniform ladder (8 quant levels × 25 prompts × 10 seeds = 2,000 images)
python generate_ladder.py --output-dir ./results/ladder

# 2. Generate asymmetric configs (4 switch points × 25 prompts × 10 seeds = 1,000 images)
python generate_asymmetric.py --ladder-dir ./results/ladder --output-dir ./results/asymmetric

# 3. Measure all metrics
python measure_lpips_clip.py    --ladder-dir ./results/ladder --asym-dir ./results/asymmetric --output-dir ./results/metrics
python measure_quality_score.py --ladder-dir ./results/ladder --asym-dir ./results/asymmetric --output-dir ./results/metrics
python measure_imagereward.py   --ladder-dir ./results/ladder --asym-dir ./results/asymmetric --output-dir ./results/metrics
python measure_fid.py           --ladder-dir ./results/ladder --asym-dir ./results/asymmetric --output-dir ./results/metrics

# 4. Speed benchmark
python benchmark_speed.py --output-dir ./results/metrics

# 5. Generate figures
python plot_metrics.py --metrics-dir ./results/metrics --output-dir ./results/figures

Pipeline Overview

generate_ladder.py          ─→  2,000 uniform images
generate_asymmetric.py      ─→  1,000 asymmetric images
                                        │
           ┌────────────────────────────┼────────────────────────────┐
           ↓                            ↓                            ↓
  measure_lpips_clip.py     measure_quality_score.py     measure_imagereward.py
  (LPIPS + CLIP Score)      (ref-free Quality Score)     (ImageReward)
           │                            │                            │
           └────────────────────────────┼────────────────────────────┘
                                        ↓
                               measure_fid.py (FID)
                                        ↓
                               plot_metrics.py (figures)

Optional: Step-by-Step Analysis

These scripts generate the intermediate latent visualizations (Figure 1 in the paper):

# Save per-step latents for each quant level
python generate_stepwise.py --output-dir ./results/stepwise

# Decode latents into panorama strips
python generate_panorama.py --stepwise-dir ./results/stepwise --output-dir ./results/panorama

# Assemble comparison grid
python plot_grid.py --panorama-dir ./results/panorama --output-dir ./results/figures

Script Reference

Generation

Script	Description	Output	Time Estimate
generate_ladder.py	Uniform quant ladder (Q2_K–Q8_0)	2,000 PNG images	~28h on RTX 4070
generate_asymmetric.py	Asymmetric configs (k=3,5,7,10)	1,000 PNG images	~14h on RTX 4070
generate_stepwise.py	Per-step latent snapshots	.pt files per step	~28h on RTX 4070
generate_panorama.py	Decode stepwise latents to image strips	PNG panoramas	~30min

Measurement

Script	Metrics	Type	Dependencies
measure_lpips_clip.py	LPIPS, CLIP Score	Reference-based + Semantic	lpips, open-clip-torch
measure_quality_score.py	Quality Score	Reference-free	open-clip-torch
measure_imagereward.py	ImageReward	Reference-free (human pref)	image-reward
measure_fid.py	FID	Distribution-based	clean-fid
benchmark_speed.py	Time, Peak VRAM	Performance	nvidia-smi in PATH

Visualization

Script	Output
plot_metrics.py	Bar charts, heatmaps, scatter plots (paper figures)
plot_grid.py	Step-by-step comparison grid (Figure 1)
plot_table.py	Table images (Table 1, 2)

File Naming Convention

All generation scripts follow a consistent naming pattern:

{prompt_id}_{config_tag}_seed{seed}_00001_.png

Examples:

• A01_Q8_0_seed42_00001_.png — uniform Q8_0
• A01_asym_Q2K_Q5_1_k3_seed42_00001_.png — asymmetric k=3

Resuming Interrupted Runs

All generation scripts support resuming. They check for existing output files and skip already-generated images. Simply re-run the same command to continue from where it stopped.

Output CSV Format

quality_lpips_clip.csv

config, prompt_id, category, seed, lpips_vs_Q8_0, clip_score

quality_reffree.csv

config, prompt_id, category, seed, clip_score, quality_score

quality_imagereward.csv

config, prompt_id, category, seed, image_reward

fid_results.csv

config, n_images, fid_vs_Q8_0

benchmark_results.csv

config, type, model_a, model_b, switch_step, run_idx, is_cold, time_sec, peak_vram_mb, status