Some tips for running the Smilei on Supercomputers

[Tissot11]

I wanted to ask for some tips based on your experiences running Smilei on supercomputers. I understand that this is hard to answer since it depends on several things. But it would be nice to have some information that I could utilise to run Smilei efficiently. I have read the parallelization page and have tried your recommendations already. In the Highlights on Smilei webpage and in other papers using Smilei, I see that hyper-threading(HT) is usually turned off. Does HT have a big influence on the Smilei runtime?

I'm running Smilei on Max-Planck supercomputers Raven and Cobra in Munich. I party had to move there because of issues related to the boundaries that I had reported in #293. I have compiled Smilei with intel compilers and intel mpi and used the vectorization flags suggested. I'm running Smilei with hyper-threading enabled and I'm seeing some inconsistent results on scaling. Running Smilei on more 10 nodes of Raven (1 node has 2 sockets and each socket has Intel Xeon 9242 CPU with 48 cores), there is a significant performance degradation, and I have experimented with with OMP_PLACES and OMP_PROC_BIND settings. I'm talking with the people in Munich to figure out a better strategy to run Smilei efficiently but if you have any tips to share based on your previous experiences then please let me know.

[mccoys]

There is no general trick to do parallelization. Each case requires some investigation.

Concerning your scaling issue, is your simulation big enough ? If it is too small, there will be not enough work to spread on all nodes.

[Tissot11]

It's a 1D simulation and it's big in the sense that I have close to 1e5 cells divided in 16384 patches and more than 100 million particles in the simulation. I also need to run it for longer durations. The simulations on 10 nodes which uses more than 95% CPU and around 10 GB RAM per node doesn't finish in 24 hours. For some reasons, simulations on restart seem to be slower as well which I need to resolve. So I would like it to get it done in less than 24 hours. Later on, I need to do 2D simulations.

[mccoys]

You can try the Performances diagnostic to try to find out the issue

[Tissot11]

Yeah, I'll run the simulations now with the Performance diagnostics enabled. Do you suggest to keep using HT or turn it off?

[mccoys]

HT off is generally better for the machines I use, but I don't know the recommendation for yours.