How to add a benchmark using BenchmarkRegistry ?

[DinizNicolas]

Hi everyone !

I am currently trying to add another line to ./superbench/benchmarks/model_benchmarks/pytorch_gpt2.py in order to run with a custom model. It is copy/pasting the line :
BenchmarkRegistry.register_benchmark( 'pytorch-gpt2-xl', PytorchGPT2, parameters='--hidden_size=1600 --num_hidden_layers=48 --num_attention_heads=25' )
To that I change the name, parameters values and add the --seq_len parameter.

Then I run the benchmark and get :
[2023-05-24 15:14:04,664 ip:563][executor.py:143][ERROR] Executor failed in gpt_models/pytorch-gpt2-13B, invalid context. [2023-05-24 15:14:04,699 ip:557][registry.py:255][WARNING] Benchmark has no implementation, name: pytorch-gpt2-13B, platform: CUDA

I searched for the place the model_benchmarks in ./superbench/benchmarks/model_benchmarks/ are added to BenchmarkRegistry dictionnary, but it was without results.

I tried to reinstall and/or redeploy with the modified file pytorch_gpt2.py without success.

I am stuck there and ran out of ideas. Any help is appreciated!

[guoshzhao]

You need to reinstall the superbench inside the sb-workspace container, otherwise the new added benchmark can't be found since the released docker image don't include your changes.