Reproducing results of "Understanding the Role of the Projector in Knowledge Distillation"

[roymiles]

I have reproduced the results in the original paper. The original paper reports an accuracy of 71.63%, while this config leads to 71.65%.

The log and checkpoint for this run can be found here: https://drive.google.com/drive/folders/18xl0CDZ6CioP4Sbjdpj1Pndp4biSLpnV?usp=sharing

Trained on a single GPU.

[yoshitomo-matsubara]

Just for consistency with existing files, could you replace 'imagenet2012' in this line with 'ilsvrc2012'?

[yoshitomo-matsubara]

Hi @roymiles

Beside the above request, I have a few more minor requests:

• Add srd-resnet18_from_resnet34.log to configs/official/ilsvrc2012/roymiles/aaai2024/
• Add your paper info to this file as the third item following the format
• Add your result to https://github.com/yoshitomo-matsubara/torchdistill/blob/main/docs/source/_static/benchmarks/imagenet-resnet18_kd.tsv
• Most important, add README.md like this file to configs/official/ilsvrc2012/roymiles/aaai2024/. You can use the links to checkpoint and log file (those links become effective when I release the next version) in your README.md file.

Thanks!

(Just for a record, roymiles/Simple-Recipe-Distillation#1)

[yoshitomo-matsubara]

Thanks @roymiles for the updates!

Did you use 3 GPUs for distributed training? If nos, I will remove the section from README later, when I make minor changes.
https://github.com/yoshitomo-matsubara/torchdistill/pull/446/files#diff-1ecd33e0a6aeb10ddebfcdc6ed245a3e8ea60e38a09ed8974047a3101ec638aeR41-R53

[roymiles]

Ah oops, I must have overlooked that. Yea I only used 1 GPU.

[yoshitomo-matsubara]

No problem, I will merge this PR and make some modifications.
The next version of torchdistill will be released in a few days, and I will upload the checkpoint and log as part of the release note for backup.

It's a great job! Thanks for your contribution!