Build SCR

Dependencies

SCR has several required dependencies. Others are optional, and if not available, corresponding SCR functionality is disabled.

Required:

  • C and C++ compilers

  • CMake, Version 3.14.5+

  • MPI 3.0+

Optional:

To simplify the install process, one can use CMake to build a release tarball or use Spack.

The CMake and Spack sections below assume that one is installing SCR on a system with existing compilers, a resource manager (like SLURM or LSF), and an MPI environment. These base software packages are typically preinstalled and configured for users by the support staff of HPC clusters.

CMake

SCR requires CMake version 3.14.5 or higher. The SCR build uses the CMake FindMPI module to link with MPI. This module looks for the standard mpicc compiler wrapper, which must be in your PATH.

One can download an SCR release tarball from the GitHub release page. To build SCR from a release tarball:

wget https://github.com/LLNL/scr/releases/download/v3.0/scr-v3.0.tgz
tar -zxf scr-v3.0.tgz
cd scr-v3.0

mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=../install ..
make -j install

Some common CMake command line options:

  • -DCMAKE_INSTALL_PREFIX=[path]: Place to install the SCR library

  • -DCMAKE_BUILD_TYPE=[Debug/Release]: Build with debugging or optimizations, defaults to Release

  • -DBUILD_SHARED_LIBS=[ON/OFF]: Whether to build shared libraries, defaults to ON

  • -DSCR_RESOURCE_MANAGER=[SLURM/FLUX/APRUN/LSF/NONE] : Resource manager for job allocations, defaults to SLURM

  • -DSCR_CNTL_BASE=[path] : Path to SCR Control directory, defaults to /dev/shm

  • -DSCR_CACHE_BASE=[path] : Path to SCR Cache directory, defaults to /dev/shm

  • -DSCR_CONFIG_FILE=[path] : Path to SCR system configuration file, defaults to <install>/etc/scr.conf

  • -DSCR_FILE_LOCK=[FLOCK/FCNTL/NONE] : Specify type of file locking to use, defaults to FLOCK

For setting the default logging parameters:

  • -DSCR_LOG_ENABLE=[0/1] : Whether to enable SCR logging of any type (1) or not (0), defaults to 0

  • -DSCR_LOG_SYSLOG_ENABLE=[0/1] : Whether to enable SCR logging via syslog (1) or not (0), defaults to 1

  • -DSCR_LOG_SYSLOG_FACILITY=[facility] : Facility for syslog messages (see man openlog), defaults to LOG_LOCAL7

  • -DSCR_LOG_SYSLOG_LEVEL=[level] : Level for syslog messages (see man openlog), defaults to LOG_INFO

  • -DSCR_LOG_SYSLOG_PREFIX=[str] : Prefix string to prepend to syslog messages, defaults to SCR

  • -DSCR_LOG_TXT_ENABLE=[0/1] : Whether to enable SCR logging to a text file (1) or not (0), defaults to 1

One can disable portions of the SCR build if they are not needed:

  • -DENABLE_FORTRAN=[ON/OFF] : Whether to build library for Fortran bindings, defaults to ON

  • -DENABLE_FORTRAN_TRAILING_UNDERSCORES=[AUTO/ON/OFF] : Whether to append underscores to symbol names in the Fortran bindings, defaults to AUTO

  • -DENABLE_EXAMPLES=[ON/OFF] : Whether to build programs in examples directory, defaults to ON

  • -DENABLE_TESTS=[ON/OFF] : Whether to support make check tests, defaults to ON

  • -DENABLE_PTHREADS=[ON/OFF] : Whether to enable pthreads support for file transfers, defaults to ON

  • -DENABLE_IBM_BBAPI=[ON/OFF] : Whether to enable IBM Burst Buffer support for file transfers, defaults to OFF

  • -DENABLE_CRAY_DW=[ON/OFF] : Whether to enable Cray DataWarp support for file transfers, defaults to OFF

  • -DENABLE_PDSH=[ON/OFF] : Whether to use pdsh to check node health and scavenge files, defalts to ON

  • -DBUILD_PDSH=[ON/OFF]: CMake can automatically download and build the PDSH dependency, defaults to OFF

  • -DWITH_PDSH_PREFIX=[path to PDSH]: Path to an existing PDSH installation (should not be used with BUILD_PDSH)

  • -DENABLE_YOGRT=[ON/OFF] : Whether to use libyogrt for determining allocation end time, defaults to ON

  • -DWITH_YOGRT_PREFIX:PATH=[path to libyogrt]

  • -DENABLE_MYSQL=[ON/OFF] : Whether to use MySQL for logging, defaults to OFF

  • -DWITH_MYSQL_PREFIX=[path to MySQL]

Spack

If you use the Spack package manager, SCR and many of its dependencies have corresponding packages.

Before installing SCR with Spack, one should first properly configure packages.yaml. In particular, SCR depends on the system resource manager and MPI library, and one should define entries for those in packages.yaml.

By default, Spack attempts to build all dependencies for SCR, including packages such as SLURM, MPI, and OpenSSL that are already installed on most HPC systems. It is recommended to use the system-installed software when possible. This ensures that the resulting SCR build actually works on the target system, and it can significantly reduce the build time.

Spack uses its packages.yaml file to locate external packages. Full information about packages.yaml can be found in the Spack documentation.

At minimum, it is important to register the system MPI library and the system resource manager. Other packages can be defined to accelerate the build. The following shows example entries for packages.yaml. One must modify these example entries to use the proper versions, module names, and paths for the target system:

packages:
  all:
    providers:
      mpi: [mvapich2,openmpi,spectrum-mpi]

  # example entry for MVAPICH2 MPI, accessed by a module named mvapich2
  mvapich2:
    buildable: false
    externals:
    - spec: mvapich2
      modules:
      - mvapich2

  # example entry for Open MPI
  openmpi:
    buildable: false
    externals:
    - spec: openmpi@4.1.0
      prefix: /opt/openmpi-4.1.0

  # example entry for IBM Spectrum MPI
  spectrum-mpi:
    buildable: false
    externals:
    - spec: spectrum-mpi
      prefix: /opt/ibm/spectrum_mpi

  # example entry for IBM LSF resource manager
  lsf:
    buildable: false
    externals:
    - spec: lsf@10.1
      prefix: /opt/ibm/spectrumcomputing/lsf/10.1

  # example entry for SLURM resource manager
  slurm:
    buildable: false
    externals:
    - spec: slurm@20
      prefix: /usr

  openssl:
    externals:
    - spec: openssl@1.0.2
      prefix: /usr

  libyogrt:
    externals:
    - spec: libyogrt scheduler=lsf
      prefix: /usr
    - spec: libyogrt scheduler=slurm
      prefix: /usr

The packages key declares the following block as a set of package descriptions. The following descriptions tell Spack how to find items that already installed on the system.

  • The providers key specifies that one of three different MPI versions are available, MVAPICH2, Open MPI, or IBM Spectrum MPI.

  • mvapich2: declares that MVAPICH2 is available, and the location is defined in a mvapich2 module file.

  • openmpi: declares that Open MPI is installed in the system at the location specified by prefix, and the buildable: false line declares that Spack should always use that version of MPI rather than try to build its own. This description addresses the common situation where MPI is customized and optimized for the local system, and Spack should never try to compile a replacement.

  • spectrum-mpi: declares that Spectrum MPI is available.

  • lsf: declares that if LSF is needed (e.g. to use scheduler=lsf) the libraries can be found at the specified prefix.

  • slurm: declares that if SLURM is needed (e.g. to use scheduler=slurm) the libraries can be found at the specified prefix.

  • openssl: declares that openssl version 1.0.2 is installed on the system and that Spack should use that if it satisfies the dependencies required by any spack-installed packages, but if a different version is requested, Spack should install its own version.

  • libyogrt: declares that libyogrt is installed, but Spack may decide to build its own version. If scheduler=slurm or scheduler=lsf is selected, use the version installed under /usr, otherwise build from scratch using the selected scheduler.

After configuring packages.yaml, one can install SCR.

For SLURM systems, SCR can be installed with:

spack install scr@3.0 resource_manager=SLURM

For LSF, systems, SCR can be installed with:

spack install scr@3.0 resource_manager=LSF

The SCR Spack package provides other variants that may be useful. To see the full list, type:

spack info scr