Yes of course, happy to share everything I have about how the training worked. Everything is in:https://github.com/agentmorris/fish-training-private/Filippo, you have access, anyone else who is interested should send me your github ID and I will add you.I use this notebook to get the environment ready for a training pass:https://github.com/agentmorris/fish-training-private/blob/main/train_cfd_rf-detr_base.pyThe last cell says "train", but that's not actually how I run training. Instead, there is a script for each training pass in this folder:https://github.com/agentmorris/fish-training-private/tree/main/training_scriptsThat's what actually did the training.Some of the small things I had to do to get the training running:
- Validation during training crashes consistently because our validation set is pretty large, which is fine, I don't need to validate during training, but I found I couldn't just not have a validation set, instead I have to have both a validation set and a test set with one image as a dummy. https://github.com/roboflow/rf-detr/issues/449
- A bunch of COCO fields that are normally considered optional and are not relevant to training were required (i.e. training crashed without them) (e.g. "supercategory", "area", "iscrowd"). I added a few things to my YOLO --> COCO converter to support these. https://megadetector.readthedocs.io/en/latest/data_management.html#module-megadetector.data_management.yolo_to_coco
- The COCO filename in each folder (train/valid/test) must be _annotations.coco.json .
- The Ultralytics framework supports "val", "valid", or "validation" as a val folder name (or not having one), RF-DETR wants it to be "valid".
- The batch size parameter is well-documented, but a little confusing, particularly wrt multi-GPU training. What really matters is the product of two parameters, batch_size and grad_accum_steps. The documentation is clear on this for single-GPU training, less so for multi-GPU training. https://rfdetr.roboflow.com/learn/train/#dataset-structure
All of these are small things, but in a few cases the error messages weren't clear, so it took a while to figure all of these out. But once training was going, it was stable and straightforward.