2--UNetModel

一个可以debug的小demo#xff1a;SD_UNet​​​​​​​

Image

提供的文生图Demo中实际传入的参数只有x、timesteps和context三个其中

[B*2,

context表示经过CLIP编码后对应的文本Promptshape:

[B*2,

self.predict_codebook_ids:return

self.id_predictor(h)else:return

self.out(h)

函数定义如下而self.time_embed()是一个MLP函数。

def

torch.exp(-math.log(max_period)

torch.arange(start0,

half).to(devicetimesteps.device)args

timesteps[:,

nn.Sequential(linear(model_channels,

time_embed_dim),nn.SiLU(),linear(time_embed_dim,

time_embed_dim),

elementwise_affineTrue)(norm2):

LayerNorm((320,),

elementwise_affineTrue)(norm3):

LayerNorm((320,),

elementwise_affineTrue)))(proj_out):

Conv2d(320,

elementwise_affineTrue)(norm2):

LayerNorm((640,),

elementwise_affineTrue)(norm3):

LayerNorm((640,),

elementwise_affineTrue)))(proj_out):

Conv2d(640,

elementwise_affineTrue)(norm2):

LayerNorm((640,),

elementwise_affineTrue)(norm3):

LayerNorm((640,),

elementwise_affineTrue)))(proj_out):

Conv2d(640,

elementwise_affineTrue)(norm2):

LayerNorm((1280,),

elementwise_affineTrue)(norm3):

LayerNorm((1280,),

elementwise_affineTrue)))(proj_out):

Conv2d(1280,

elementwise_affineTrue)(norm2):

LayerNorm((1280,),

elementwise_affineTrue)(norm3):

LayerNorm((1280,),

elementwise_affineTrue)))(proj_out):

Conv2d(1280,

TimestepEmbedSequential(nn.Sequential,

TimestepBlock):A

nn.ModuleList([TimestepEmbedSequential(conv_nd(dims,

in_channels,

Module1和Module2的结构相同都由一个ResBlock和一个SpatialTransformer组成

init

[ResBlock(ch,time_embed_dim,dropout,out_channelsmult

model_channels,dimsdims,use_checkpointuse_checkpoint,use_scale_shift_normuse_scale_shift_norm,)]ch

mult

num_head_channelslayers.append(AttentionBlock(ch,use_checkpointuse_checkpoint,num_headsnum_heads,num_head_channelsdim_head,use_new_attention_orderuse_new_attention_order,)

not

context_dimcontext_dim))self.input_blocks.append(TimestepEmbedSequential(*layers))self._feature_size

chinput_block_chans.append(ch)#

self.input_blocks[1]

elementwise_affineTrue)(norm2):

LayerNorm((320,),

elementwise_affineTrue)(norm3):

LayerNorm((320,),

elementwise_affineTrue)))(proj_out):

Conv2d(320,

elementwise_affineTrue)(norm2):

LayerNorm((320,),

elementwise_affineTrue)(norm3):

LayerNorm((320,),

elementwise_affineTrue)))(proj_out):

Conv2d(320,

chself.input_blocks.append(TimestepEmbedSequential(ResBlock(ch,time_embed_dim,dropout,out_channelsout_ch,dimsdims,use_checkpointuse_checkpoint,use_scale_shift_normuse_scale_shift_norm,downTrue,)if

resblock_updownelse

2-3-4--Module4、Module5、Module7和Module8

与Module1和Module2的结构相同都由一个ResBlock和一个SpatialTransformer组成只有特征维度上的区别

Module10和Module12的结构相同只由一个ResBlock组成。

Sequential((0):