单卡训练语句#xff08;正常运行#xff09;#xff1a;

python

需要通过torch.distributed.launch来启动#xff0c;一般是单节点#xff0c;其中CUDA_VISIBLE…多卡训练最近在跑yolov10版本的RT-DETR用来进行目标检测。

单卡训练语句正常运行

需要通过torch.distributed.launch来启动一般是单节点其中CUDA_VISIBLE_DEVICES设置用的显卡编号也可以不用直接在main.py里面指定device也行–nproc_pre_node

python

main.pyCUDA_VISIBLE_DEVICES0,6,7

python

main.py但是运行多卡训练之后会报错有的时候训练进程会卡住。

错误信息如下

[rank0]:

/home/zyy23/yolov10/run_detr.py,

line

/home/zyy23/yolov10/ultralytics/engine/model.py,

line

/home/zyy23/yolov10/ultralytics/engine/trainer.py,

line

/home/zyy23/yolov10/ultralytics/engine/trainer.py,

line

/home/zyy23/anaconda3/envs/mypytorch_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py,

line

/home/zyy23/anaconda3/envs/mypytorch_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py,

line

/home/zyy23/anaconda3/envs/mypytorch_3.9/lib/python3.9/site-packages/torch/nn/parallel/distributed.py,

line

/home/zyy23/anaconda3/envs/mypytorch_3.9/lib/python3.9/site-packages/torch/nn/parallel/distributed.py,

line

self.reducer._rebuild_buckets():

[rank0]:

torch.nn.parallel.DistributedDataParallel,

and

model.28.dec_bbox_head.5.layers.2.bias,

model.28.dec_bbox_head.5.layers.2.weight,

model.28.dec_bbox_head.5.layers.1.bias,

model.28.dec_bbox_head.5.layers.1.weight,

model.28.dec_bbox_head.5.layers.0.bias,

model.28.dec_bbox_head.5.layers.0.weight,

model.28.dec_bbox_head.4.layers.2.bias,

model.28.dec_bbox_head.4.layers.2.weight,

model.28.dec_bbox_head.4.layers.1.bias,

model.28.dec_bbox_head.4.layers.1.weight,

model.28.dec_bbox_head.4.layers.0.bias,

model.28.dec_bbox_head.4.layers.0.weight,

model.28.dec_bbox_head.3.layers.2.bias,

model.28.dec_bbox_head.3.layers.2.weight,

model.28.dec_bbox_head.3.layers.1.bias,

model.28.dec_bbox_head.3.layers.1.weight,

model.28.dec_bbox_head.3.layers.0.bias,

model.28.dec_bbox_head.3.layers.0.weight,

model.28.dec_bbox_head.2.layers.2.bias,

model.28.dec_bbox_head.2.layers.2.weight,

model.28.dec_bbox_head.2.layers.1.bias,

model.28.dec_bbox_head.2.layers.1.weight,

model.28.dec_bbox_head.2.layers.0.bias,

model.28.dec_bbox_head.2.layers.0.weight,

model.28.dec_bbox_head.1.layers.2.bias,

model.28.dec_bbox_head.1.layers.2.weight,

model.28.dec_bbox_head.1.layers.1.bias,

model.28.dec_bbox_head.1.layers.1.weight,

model.28.dec_bbox_head.1.layers.0.bias,

model.28.dec_bbox_head.1.layers.0.weight,

model.28.dec_bbox_head.0.layers.2.bias,

model.28.dec_bbox_head.0.layers.2.weight,

model.28.dec_bbox_head.0.layers.1.bias,

model.28.dec_bbox_head.0.layers.1.weight,

model.28.dec_bbox_head.0.layers.0.bias,

model.28.dec_bbox_head.0.layers.0.weight,

model.28.enc_bbox_head.layers.2.bias,

model.28.enc_bbox_head.layers.2.weight,

model.28.enc_bbox_head.layers.1.bias,

model.28.enc_bbox_head.layers.1.weight,

model.28.enc_bbox_head.layers.0.bias,

model.28.enc_bbox_head.layers.0.weight,

model.28.denoising_class_embed.weight

[rank0]:

/home/zyy23/yolov10/run_detr.py,

line

/home/zyy23/yolov10/ultralytics/engine/model.py,

line

/home/zyy23/yolov10/ultralytics/engine/trainer.py,

line

/home/zyy23/yolov10/ultralytics/engine/trainer.py,

line

self.scaler.scale(self.loss).backward()

[rank1]:

/home/zyy23/anaconda3/envs/mypytorch_3.9/lib/python3.9/site-packages/torch/_tensor.py,

line

/home/zyy23/anaconda3/envs/mypytorch_3.9/lib/python3.9/site-packages/torch/autograd/__init__.py,

line

/home/zyy23/anaconda3/envs/mypytorch_3.9/lib/python3.9/site-packages/torch/autograd/graph.py,

line

Variable._execution_engine.run_backward(

Calls

[../third_party/gloo/gloo/transport/tcp/pair.cc:534]

Connection

/home/zyy23/yolov10/run_detr.py,

line

/home/zyy23/yolov10/ultralytics/engine/model.py,

line

/home/zyy23/yolov10/ultralytics/engine/trainer.py,

line

/home/zyy23/yolov10/ultralytics/engine/trainer.py,

line

self.scaler.scale(self.loss).backward()

[rank2]:

/home/zyy23/anaconda3/envs/mypytorch_3.9/lib/python3.9/site-packages/torch/_tensor.py,

line

/home/zyy23/anaconda3/envs/mypytorch_3.9/lib/python3.9/site-packages/torch/autograd/__init__.py,

line

/home/zyy23/anaconda3/envs/mypytorch_3.9/lib/python3.9/site-packages/torch/autograd/graph.py,

line

Variable._execution_engine.run_backward(

Calls

[../third_party/gloo/gloo/transport/tcp/pair.cc:534]

Connection

torch/distributed/elastic/multiprocessing/api.py:858]

Sending

torch/distributed/elastic/multiprocessing/api.py:858]

Sending

torch/distributed/elastic/multiprocessing/api.py:833]

failed

/home/zyy23/anaconda3/envs/mypytorch_3.9/bin/python

Traceback

/home/zyy23/anaconda3/envs/mypytorch_3.9/lib/python3.9/runpy.py,

line

/home/zyy23/anaconda3/envs/mypytorch_3.9/lib/python3.9/runpy.py,

line

/home/zyy23/anaconda3/envs/mypytorch_3.9/lib/python3.9/site-packages/torch/distributed/run.py,

line

/home/zyy23/anaconda3/envs/mypytorch_3.9/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py,

line

/home/zyy23/anaconda3/envs/mypytorch_3.9/lib/python3.9/site-packages/torch/distributed/run.py,

line

/home/zyy23/anaconda3/envs/mypytorch_3.9/lib/python3.9/site-packages/torch/distributed/run.py,

line

/home/zyy23/anaconda3/envs/mypytorch_3.9/lib/python3.9/site-packages/torch/distributed/launcher/api.py,

line

/home/zyy23/anaconda3/envs/mypytorch_3.9/lib/python3.9/site-packages/torch/distributed/launcher/api.py,

line

torch.distributed.elastic.multiprocessing.errors.ChildFailedError:run_detr.py

FAILED

------------------------------------------------------------

------------------------------------------------------------

Root

https://pytorch.org/docs/stable/elastic/errors.html发生了runtimerror

RuntimeError:

torch.nn.parallel.DistributedDataParallel,

and

运行时错误预计在开始新迭代之前已完成前一次迭代的减少。

此错误表明您的模块具有未用于产生损耗的参数。

您可以通过

(1)

torch.nn.parallel.DistributedDataParallel

(2)

函数输出都参与计算损失。

如果您已经完成了上述两个步骤那么分布式数据并行模块无法在模块的

forward

torch.nn.parallel.DistributedDataParallel中加入find_unused_parameters参数并设置初始值为Truefind_unused_parameters

PyTorch

nn.parallel.DistributedDataParallel(self.model,

device_ids[RANK],find_unused_parametersTrue)find_unused_parameters参数的作用

True

会在每个前向传播过程中检查哪些参数没有被使用。

这对于某些模型来说非常有用特别是当模型的某些部分在特定输入下可能不会被触发时。

减少损失计算时的内存开销提高性能通过识别并忽略未使用的参数PyTorch

可以在计算梯度时减少内存开销从而提高训练效率。

在大型模型或复杂网络中识别未使用的参数可以加速训练过程尤其是在分布式设置中因为可以避免不必要的梯度同步。

但是它会增加额外的计算开销用于验证哪些参数是未使用的不用参加到损失的计算中所以最好是仅在需要时使用此参数尤其是在模型中确实存在未使用参数的情况下。

由于用的是yolov10封装的太严密一直找不到这条语句在哪个位置找了好久也尝试在模型初始化的位置和命令行加参数都没成功后来在ultralytics/engine里面的trainer.py找到了在_setup_train函数下面。

def

Modelself.run_callbacks(on_pretrain_routine_start)ckpt

self.model.to(self.device)self.set_model_attributes()#

Freeze

self.model.named_parameters():#

torch.nan_to_num(x))

freeze_layer_names):LOGGER.info(fFreezing

layer

torch.tensor(self.args.amp).to(self.device)

True

callbacks.default_callbacks.copy()

backup

torch.tensor(check_amp(self.model),

deviceself.device)callbacks.default_callbacks

callbacks_backup

torch.cuda.amp.GradScaler(enabledself.amp)if

world_size

nn.parallel.DistributedDataParallel(self.model,

device_ids[RANK],find_unused_parametersTrue)二是将环境变量TORCH_DISTRIBUTED_DEBUG设置为INFO或DETAIL打印有关哪些特定参数在此级别上没有收到梯度的信息作为此错误的一部分。

上述报错信息没有显示这个提示是因为我们已经使用了这条语句使用之后会看到哪些参数没有收到梯度信息。

python

self.model.named_parameters():if

param.grad

is:)print(name)在使用nn.Module的__init__方法时如果使用self.xx这样的语句定义了层但是这个层的计算结果后续没有用来计算loss或者这个self层没有使用都会导致报错。

只需要在模型中仔细检查forward函数和init函数检查__init__()

forward