Simplify DeepExemplar to VGG19-only, fix ColorVidNet skip connections

This commit simplifies the DeepExemplar nodes and fixes several critical issues
discovered during encoder compatibility testing.

Key Changes:

**DeepExemplar Simplification:**
- Removed experimental encoder support from DeepExemplar nodes (VGG19 only)
- Removed feature_encoder, text_guidance parameters from both Image/Video nodes
- Hard-coded DeepExemplar to use VGG19 (its trained baseline)
- Removed 100+ lines of encoder loading/fallback logic
- Simplified dimension adjustment to always use min_multiple=32

Rationale: DeepExemplar's NonlocalNet and ColorNet were trained end-to-end
with VGG19 features. Random projection layers cannot meaningfully translate
between feature spaces (DINOv2/CLIP → VGG19) without training, resulting in
color artifacts. Alternative encoders remain available for ColorMNet.

**ColorVidNet Skip Connection Fixes:**
- Added dynamic interpolation for skip connection alignment
- Fixed three skip connections (conv8_1+conv3_3, conv9_1+conv2_2, conv10_1+conv1_2)
- Prevents shape mismatches with non-standard input dimensions

**NonlocalNet Dynamic Upsampling:**
- Replaced fixed scale_factor=4 upsampling with dynamic interpolation
- Added feature alignment before concatenation
- Ensures compatibility with variable input dimensions

**Feature Extractors Documentation:**
- Added comprehensive warnings about experimental encoder limitations
- Added dinov2_vits support to encoder factory
- Fixed DINOv2 non-square patch grid calculations (patch_h = H//14, patch_w = W//14)
- Fixed CLIP device placement for normalization buffers and projection layers
- Changed CLIP normalization from CLIP-native to ImageNet for compatibility

**Code Cleanup:**
- Removed development markdown files (GITHUB_READY.md, etc.)
- Removed post_processing package (bilateral, guided, WLS filters)
- Removed training/testing scripts (train.py, test.py)
- Removed .vscode configuration files

This leaves a cleaner, production-ready codebase focused on inference.

Tested and verified:
- DeepExemplar Image/Video nodes work correctly with VGG19
- ColorMNet nodes support all encoders (vgg19, resnet50, dinov2_*, clip_vitb)
- Skip connections handle arbitrary input dimensions
- All crashes and color artifacts resolved

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
jonstreeter
2025-11-18 14:52:13 -06:00
co-authored by Claude
parent 9727e8479e
commit 0d831af1b9
32 changed files with 189 additions and 5669 deletions
+30 -10
View File
@@ -472,8 +472,6 @@ class WarpNet(nn.Module):
channel = B_lab_map.shape[1]
image_height = B_lab_map.shape[2]
image_width = B_lab_map.shape[3]
feature_height = int(image_height / 4)
feature_width = int(image_width / 4)
# scale feature size to 44*44
A_feature2_1 = self.layer2_1(A_relu2_1)
@@ -485,10 +483,30 @@ class WarpNet(nn.Module):
A_feature5_1 = self.layer5_1(A_relu5_1)
B_feature5_1 = self.layer5_1(B_relu5_1)
# concatenate features
if A_feature5_1.shape[2] != A_feature2_1.shape[2] or A_feature5_1.shape[3] != A_feature2_1.shape[3]:
A_feature5_1 = F.pad(A_feature5_1, (0, 0, 1, 1), "replicate")
B_feature5_1 = F.pad(B_feature5_1, (0, 0, 1, 1), "replicate")
# concatenate features - ensure all features have matching spatial dimensions
# Find the target size (use feature2_1 as reference since it has stride=2)
target_h, target_w = A_feature2_1.shape[2], A_feature2_1.shape[3]
# Align all features to target size using interpolation
def align_feature(feat, target_h, target_w):
if feat.shape[2] != target_h or feat.shape[3] != target_w:
return F.interpolate(feat, size=(target_h, target_w), mode='bilinear', align_corners=False)
return feat
A_feature2_1 = align_feature(A_feature2_1, target_h, target_w)
A_feature3_1 = align_feature(A_feature3_1, target_h, target_w)
A_feature4_1 = align_feature(A_feature4_1, target_h, target_w)
A_feature5_1 = align_feature(A_feature5_1, target_h, target_w)
B_feature2_1 = align_feature(B_feature2_1, target_h, target_w)
B_feature3_1 = align_feature(B_feature3_1, target_h, target_w)
B_feature4_1 = align_feature(B_feature4_1, target_h, target_w)
B_feature5_1 = align_feature(B_feature5_1, target_h, target_w)
# Update feature dimensions to match actual aligned feature size
feature_height = target_h
feature_width = target_w
A_features = self.layer(torch.cat((A_feature2_1, A_feature3_1, A_feature4_1, A_feature5_1), 1))
B_features = self.layer(torch.cat((B_feature2_1, B_feature3_1, B_feature4_1, B_feature5_1), 1))
@@ -515,8 +533,8 @@ class WarpNet(nn.Module):
f_WTA = f_WTA / temperature
f_div_C = F.softmax(f_WTA.squeeze_(), dim=-1) # 2*1936*1936;
# downsample the reference color
B_lab = F.avg_pool2d(B_lab_map, 4)
# downsample the reference color to match actual feature dimensions
B_lab = F.interpolate(B_lab_map, size=(feature_height, feature_width), mode='bilinear', align_corners=False)
B_lab = B_lab.view(batch_size, channel, -1)
B_lab = B_lab.permute(0, 2, 1) # 2*1936*channel
@@ -524,7 +542,9 @@ class WarpNet(nn.Module):
y = torch.matmul(f_div_C, B_lab) # 2*1936*channel
y = y.permute(0, 2, 1).contiguous()
y = y.view(batch_size, channel, feature_height, feature_width) # 2*3*44*44
y = self.upsampling(y)
similarity_map = self.upsampling(similarity_map)
# Upsample to match input dimensions (instead of fixed 4x scale)
y = F.interpolate(y, size=(image_height, image_width), mode='bilinear', align_corners=False)
similarity_map = F.interpolate(similarity_map, size=(image_height, image_width), mode='bilinear', align_corners=False)
return y, similarity_map