Simplify DeepExemplar to VGG19-only, fix ColorVidNet skip connections
This commit simplifies the DeepExemplar nodes and fixes several critical issues discovered during encoder compatibility testing. Key Changes: **DeepExemplar Simplification:** - Removed experimental encoder support from DeepExemplar nodes (VGG19 only) - Removed feature_encoder, text_guidance parameters from both Image/Video nodes - Hard-coded DeepExemplar to use VGG19 (its trained baseline) - Removed 100+ lines of encoder loading/fallback logic - Simplified dimension adjustment to always use min_multiple=32 Rationale: DeepExemplar's NonlocalNet and ColorNet were trained end-to-end with VGG19 features. Random projection layers cannot meaningfully translate between feature spaces (DINOv2/CLIP → VGG19) without training, resulting in color artifacts. Alternative encoders remain available for ColorMNet. **ColorVidNet Skip Connection Fixes:** - Added dynamic interpolation for skip connection alignment - Fixed three skip connections (conv8_1+conv3_3, conv9_1+conv2_2, conv10_1+conv1_2) - Prevents shape mismatches with non-standard input dimensions **NonlocalNet Dynamic Upsampling:** - Replaced fixed scale_factor=4 upsampling with dynamic interpolation - Added feature alignment before concatenation - Ensures compatibility with variable input dimensions **Feature Extractors Documentation:** - Added comprehensive warnings about experimental encoder limitations - Added dinov2_vits support to encoder factory - Fixed DINOv2 non-square patch grid calculations (patch_h = H//14, patch_w = W//14) - Fixed CLIP device placement for normalization buffers and projection layers - Changed CLIP normalization from CLIP-native to ImageNet for compatibility **Code Cleanup:** - Removed development markdown files (GITHUB_READY.md, etc.) - Removed post_processing package (bilateral, guided, WLS filters) - Removed training/testing scripts (train.py, test.py) - Removed .vscode configuration files This leaves a cleaner, production-ready codebase focused on inference. Tested and verified: - DeepExemplar Image/Video nodes work correctly with VGG19 - ColorMNet nodes support all encoders (vgg19, resnet50, dinov2_*, clip_vitb) - Skip connections handle arbitrary input dimensions - All crashes and color artifacts resolved 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
+30
-10
@@ -472,8 +472,6 @@ class WarpNet(nn.Module):
|
||||
channel = B_lab_map.shape[1]
|
||||
image_height = B_lab_map.shape[2]
|
||||
image_width = B_lab_map.shape[3]
|
||||
feature_height = int(image_height / 4)
|
||||
feature_width = int(image_width / 4)
|
||||
|
||||
# scale feature size to 44*44
|
||||
A_feature2_1 = self.layer2_1(A_relu2_1)
|
||||
@@ -485,10 +483,30 @@ class WarpNet(nn.Module):
|
||||
A_feature5_1 = self.layer5_1(A_relu5_1)
|
||||
B_feature5_1 = self.layer5_1(B_relu5_1)
|
||||
|
||||
# concatenate features
|
||||
if A_feature5_1.shape[2] != A_feature2_1.shape[2] or A_feature5_1.shape[3] != A_feature2_1.shape[3]:
|
||||
A_feature5_1 = F.pad(A_feature5_1, (0, 0, 1, 1), "replicate")
|
||||
B_feature5_1 = F.pad(B_feature5_1, (0, 0, 1, 1), "replicate")
|
||||
# concatenate features - ensure all features have matching spatial dimensions
|
||||
# Find the target size (use feature2_1 as reference since it has stride=2)
|
||||
target_h, target_w = A_feature2_1.shape[2], A_feature2_1.shape[3]
|
||||
|
||||
# Align all features to target size using interpolation
|
||||
def align_feature(feat, target_h, target_w):
|
||||
if feat.shape[2] != target_h or feat.shape[3] != target_w:
|
||||
return F.interpolate(feat, size=(target_h, target_w), mode='bilinear', align_corners=False)
|
||||
return feat
|
||||
|
||||
A_feature2_1 = align_feature(A_feature2_1, target_h, target_w)
|
||||
A_feature3_1 = align_feature(A_feature3_1, target_h, target_w)
|
||||
A_feature4_1 = align_feature(A_feature4_1, target_h, target_w)
|
||||
A_feature5_1 = align_feature(A_feature5_1, target_h, target_w)
|
||||
|
||||
B_feature2_1 = align_feature(B_feature2_1, target_h, target_w)
|
||||
B_feature3_1 = align_feature(B_feature3_1, target_h, target_w)
|
||||
B_feature4_1 = align_feature(B_feature4_1, target_h, target_w)
|
||||
B_feature5_1 = align_feature(B_feature5_1, target_h, target_w)
|
||||
|
||||
# Update feature dimensions to match actual aligned feature size
|
||||
feature_height = target_h
|
||||
feature_width = target_w
|
||||
|
||||
A_features = self.layer(torch.cat((A_feature2_1, A_feature3_1, A_feature4_1, A_feature5_1), 1))
|
||||
B_features = self.layer(torch.cat((B_feature2_1, B_feature3_1, B_feature4_1, B_feature5_1), 1))
|
||||
|
||||
@@ -515,8 +533,8 @@ class WarpNet(nn.Module):
|
||||
f_WTA = f_WTA / temperature
|
||||
f_div_C = F.softmax(f_WTA.squeeze_(), dim=-1) # 2*1936*1936;
|
||||
|
||||
# downsample the reference color
|
||||
B_lab = F.avg_pool2d(B_lab_map, 4)
|
||||
# downsample the reference color to match actual feature dimensions
|
||||
B_lab = F.interpolate(B_lab_map, size=(feature_height, feature_width), mode='bilinear', align_corners=False)
|
||||
B_lab = B_lab.view(batch_size, channel, -1)
|
||||
B_lab = B_lab.permute(0, 2, 1) # 2*1936*channel
|
||||
|
||||
@@ -524,7 +542,9 @@ class WarpNet(nn.Module):
|
||||
y = torch.matmul(f_div_C, B_lab) # 2*1936*channel
|
||||
y = y.permute(0, 2, 1).contiguous()
|
||||
y = y.view(batch_size, channel, feature_height, feature_width) # 2*3*44*44
|
||||
y = self.upsampling(y)
|
||||
similarity_map = self.upsampling(similarity_map)
|
||||
|
||||
# Upsample to match input dimensions (instead of fixed 4x scale)
|
||||
y = F.interpolate(y, size=(image_height, image_width), mode='bilinear', align_corners=False)
|
||||
similarity_map = F.interpolate(similarity_map, size=(image_height, image_width), mode='bilinear', align_corners=False)
|
||||
|
||||
return y, similarity_map
|
||||
|
||||
Reference in New Issue
Block a user