Build SCR
Dependencies
SCR has several required dependencies. Others are optional, and if not available, corresponding SCR functionality is disabled.
Required:
C and C++ compilers
CMake, Version 3.14.5+
MPI 3.0+
Optional:
Fortran compiler (for Fortran bindings)
pdsh (for scalable restart and scavenge) (https://github.com/chaos/pdsh)
libyogrt (for time remaining in a job allocation) (https://github.com/llnl/libyogrt)
MySQL (for logging SCR activities)
To simplify the install process, one can use CMake to build a release tarball or use Spack.
The CMake and Spack sections below assume that one is installing SCR on a system with existing compilers, a resource manager (like SLURM or LSF), and an MPI environment. These base software packages are typically preinstalled and configured for users by the support staff of HPC clusters.
CMake
SCR requires CMake version 3.14.5 or higher.
The SCR build uses the CMake FindMPI module to link with MPI.
This module looks for the standard mpicc
compiler wrapper,
which must be in your PATH
.
One can download an SCR release tarball from the GitHub release page. To build SCR from a release tarball:
wget https://github.com/LLNL/scr/releases/download/v3.0/scr-v3.0.tgz
tar -zxf scr-v3.0.tgz
cd scr-v3.0
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=../install ..
make -j install
Some common CMake command line options:
-DCMAKE_INSTALL_PREFIX=[path]
: Place to install the SCR library-DCMAKE_BUILD_TYPE=[Debug/Release]
: Build with debugging or optimizations, defaults toRelease
-DBUILD_SHARED_LIBS=[ON/OFF]
: Whether to build shared libraries, defaults toON
-DSCR_RESOURCE_MANAGER=[SLURM/FLUX/APRUN/LSF/NONE]
: Resource manager for job allocations, defaults toSLURM
-DSCR_CNTL_BASE=[path]
: Path to SCR Control directory, defaults to/dev/shm
-DSCR_CACHE_BASE=[path]
: Path to SCR Cache directory, defaults to/dev/shm
-DSCR_CONFIG_FILE=[path]
: Path to SCR system configuration file, defaults to<install>/etc/scr.conf
-DSCR_FILE_LOCK=[FLOCK/FCNTL/NONE]
: Specify type of file locking to use, defaults toFLOCK
For setting the default logging parameters:
-DSCR_LOG_ENABLE=[0/1]
: Whether to enable SCR logging of any type (1) or not (0), defaults to0
-DSCR_LOG_SYSLOG_ENABLE=[0/1]
: Whether to enable SCR logging via syslog (1) or not (0), defaults to1
-DSCR_LOG_SYSLOG_FACILITY=[facility]
: Facility for syslog messages (see man openlog), defaults toLOG_LOCAL7
-DSCR_LOG_SYSLOG_LEVEL=[level]
: Level for syslog messages (see man openlog), defaults toLOG_INFO
-DSCR_LOG_SYSLOG_PREFIX=[str]
: Prefix string to prepend to syslog messages, defaults toSCR
-DSCR_LOG_TXT_ENABLE=[0/1]
: Whether to enable SCR logging to a text file (1) or not (0), defaults to1
One can disable portions of the SCR build if they are not needed:
-DENABLE_FORTRAN=[ON/OFF]
: Whether to build library for Fortran bindings, defaults toON
-DENABLE_FORTRAN_TRAILING_UNDERSCORES=[AUTO/ON/OFF]
: Whether to append underscores to symbol names in the Fortran bindings, defaults toAUTO
-DENABLE_EXAMPLES=[ON/OFF]
: Whether to build programs inexamples
directory, defaults toON
-DENABLE_TESTS=[ON/OFF]
: Whether to supportmake check
tests, defaults toON
-DENABLE_PTHREADS=[ON/OFF]
: Whether to enable pthreads support for file transfers, defaults toON
-DENABLE_IBM_BBAPI=[ON/OFF]
: Whether to enable IBM Burst Buffer support for file transfers, defaults toOFF
-DENABLE_CRAY_DW=[ON/OFF]
: Whether to enable Cray DataWarp support for file transfers, defaults toOFF
-DENABLE_PDSH=[ON/OFF]
: Whether to use pdsh to check node health and scavenge files, defalts toON
-DBUILD_PDSH=[ON/OFF]
: CMake can automatically download and build the PDSH dependency, defaults toOFF
-DWITH_PDSH_PREFIX=[path to PDSH]
: Path to an existing PDSH installation (should not be used withBUILD_PDSH
)-DENABLE_YOGRT=[ON/OFF]
: Whether to use libyogrt for determining allocation end time, defaults toON
-DWITH_YOGRT_PREFIX:PATH=[path to libyogrt]
-DENABLE_MYSQL=[ON/OFF]
: Whether to use MySQL for logging, defaults toOFF
-DWITH_MYSQL_PREFIX=[path to MySQL]
Spack
If you use the Spack package manager, SCR and many of its dependencies have corresponding packages.
Before installing SCR with Spack,
one should first properly configure packages.yaml
.
In particular, SCR depends on the system resource manager and MPI library,
and one should define entries for those in packages.yaml
.
By default, Spack attempts to build all dependencies for SCR, including packages such as SLURM, MPI, and OpenSSL that are already installed on most HPC systems. It is recommended to use the system-installed software when possible. This ensures that the resulting SCR build actually works on the target system, and it can significantly reduce the build time.
Spack uses its packages.yaml
file to locate external packages.
Full information about packages.yaml
can be found
in the Spack documentation.
At minimum, it is important to register the system MPI library and the system resource manager.
Other packages can be defined to accelerate the build.
The following shows example entries for packages.yaml
.
One must modify these example entries to use the proper versions,
module names, and paths for the target system:
packages:
all:
providers:
mpi: [mvapich2,openmpi,spectrum-mpi]
# example entry for MVAPICH2 MPI, accessed by a module named mvapich2
mvapich2:
buildable: false
externals:
- spec: mvapich2
modules:
- mvapich2
# example entry for Open MPI
openmpi:
buildable: false
externals:
- spec: openmpi@4.1.0
prefix: /opt/openmpi-4.1.0
# example entry for IBM Spectrum MPI
spectrum-mpi:
buildable: false
externals:
- spec: spectrum-mpi
prefix: /opt/ibm/spectrum_mpi
# example entry for IBM LSF resource manager
lsf:
buildable: false
externals:
- spec: lsf@10.1
prefix: /opt/ibm/spectrumcomputing/lsf/10.1
# example entry for SLURM resource manager
slurm:
buildable: false
externals:
- spec: slurm@20
prefix: /usr
openssl:
externals:
- spec: openssl@1.0.2
prefix: /usr
libyogrt:
externals:
- spec: libyogrt scheduler=lsf
prefix: /usr
- spec: libyogrt scheduler=slurm
prefix: /usr
The packages key declares the following block as a set of package descriptions. The following descriptions tell Spack how to find items that already installed on the system.
The providers key specifies that one of three different MPI versions are available, MVAPICH2, Open MPI, or IBM Spectrum MPI.
mvapich2
: declares that MVAPICH2 is available, and the location is defined in a mvapich2 module file.openmpi
: declares that Open MPI is installed in the system at the location specified by prefix, and the buildable: false line declares that Spack should always use that version of MPI rather than try to build its own. This description addresses the common situation where MPI is customized and optimized for the local system, and Spack should never try to compile a replacement.spectrum-mpi
: declares that Spectrum MPI is available.lsf
: declares that if LSF is needed (e.g. to use scheduler=lsf) the libraries can be found at the specified prefix.slurm
: declares that if SLURM is needed (e.g. to use scheduler=slurm) the libraries can be found at the specified prefix.openssl
: declares that openssl version 1.0.2 is installed on the system and that Spack should use that if it satisfies the dependencies required by any spack-installed packages, but if a different version is requested, Spack should install its own version.libyogrt
: declares that libyogrt is installed, but Spack may decide to build its own version. If scheduler=slurm or scheduler=lsf is selected, use the version installed under /usr, otherwise build from scratch using the selected scheduler.
After configuring packages.yaml
, one can install SCR.
For SLURM systems, SCR can be installed with:
spack install scr@3.0 resource_manager=SLURM
For LSF, systems, SCR can be installed with:
spack install scr@3.0 resource_manager=LSF
The SCR Spack package provides other variants that may be useful. To see the full list, type:
spack info scr