Compare local and weekly benchmarks using Hatchet

[chapman39]

• create script to compare benchmarks of a local build and weekly shared benchmarks on LC
• improve handling of cmake build type in config-build.py
• create optional, manual CI pipeline (ruby-gcc, ruby-clang, lassen-clang) to test current PR
• documentation on how to use this script and run the manual CI pipeline

tmp todo:

• run benchmarks in a separate pipeline and have them all point at one compare call (since lassen cannot run hatchet)

How the script works

The script matches two caliper files (one from weekly shared location /usr/workspace/smithdev/califiles/serac, one from a specified build location), and creates a Hatchet "graph frame" from the difference between these two files. If the maximum difference between any section of the graph is greater than X seconds, that benchmark will "fail." The script will do this for each benchmark.

Example

../scripts/llnl/compare_benchmarks.py --current-cali-dir . --verbose --depth 2 --metric-columns "Max time/rank (inc)"

(not all graphs are shown)

You can now see the baseline and current benchmark times, as well as the difference between the two. You can also choose which "metric column" you want to see (defaults to average time per rank) and set the level of depth of the tree you wish to view. At the moment, it only displays the difference trees.

Some problems

LC system performance is inconsistent. You can run the same benchmark multiple times and get wildly different results. My understanding is this is due to the node(s) you get allocated, how busy the machine is, among other things. That being said, while this is a nice feature to look at, I'm skeptical to make this a required CI check at this time.

Improving config-build.py

Before this PR, if you set -DCMAKE_BUILD_TYPE=Release when configuring Serac, the build directory will incorrectly have debug in the name, since the args.buildtype variable remained Debug. This PR updates args.buildtype based on -DCMAKE_BUILD_TYPE, if set - and assuming --build-type option hasn't been set to anything else.

Links

• docs https://serac.readthedocs.io/en/feature-chapman39-hatchet/sphinx/dev_guide/profiling.html
• comparison ci example https://lc.llnl.gov/gitlab/smith/serac/-/pipelines/503794

[chapman39]

comparison ci https://lc.llnl.gov/gitlab/smith/serac/-/pipelines/503794