Application Development Environment¶
The application development environment on Cirrus is primarily controlled through the modules environment. By loading and switching modules you control the compilers, libraries and software available.
This means that for compiling on Cirrus you typically set the compiler you wish to use using the appropriate modules, then load all the required library modules (e.g. numerical libraries, IO format libraries).
Additionally, if you are compiling parallel applications using MPI (or SHMEM, etc.) then you will need to load one of the MPI environments and use the appropriate compiler wrapper scripts.
By default, all users on Cirrus start with no modules loaded.
Basic usage of the module
command on Cirrus is covered below. For
full documentation please see:
Using the modules environment¶
Information on the available modules¶
Finding out which modules (and hence which compilers, libraries and
software) are available on the system is performed using the
module avail
command:
[user@cirrus-login0 ~]$ module avail
...
This will list all the names and versions of the modules available on the service. Not all of them may work in your account though due to, for example, licencing restrictions. You will notice that for many modules we have more than one version, each of which is identified by a version number. One of these versions is the default. As the service develops the default version will change.
You can list all the modules of a particular type by providing an
argument to the module avail
command. For example, to list all
available versions of the Intel Compiler type:
[user@cirrus-login0 ~]$ module avail intel-compilers
--------------------------------- /mnt/lustre/indy2lfs/sw/modulefiles --------------------------------
intel-compilers-18/18.05.274 intel-compilers-19/19.0.0.117
If you want more info on any of the modules, you can use the
module help
command:
[user@cirrus-login0 ~]$ module help mpt
-------------------------------------------------------------------
Module Specific Help for /usr/share/Modules/modulefiles/mpt/2.25:
The HPE Message Passing Toolkit (MPT) is an optimized MPI
implementation for HPE systems and clusters. See the
MPI(1) man page and the MPT User's Guide for more
information.
-------------------------------------------------------------------
The simple module list
command will give the names of the modules
and their versions you have presently loaded in your environment, e.g.:
[user@cirrus-login0 ~]$ module list
Currently Loaded Modulefiles:
1) git/2.35.1(default) 6) gcc/8.2.0(default)
2) singularity/3.7.2(default) 7) intel-cc-18/18.0.5.274
3) epcc/utils 8) intel-fc-18/18.0.5.274
4) /mnt/lustre/indy2lfs/sw/modulefiles/epcc/setup-env 9) intel-compilers-18/18.05.274
5) intel-license 10) mpt/2.25
Loading, unloading and swapping modules¶
To load a module to use module add
or module load
. For example,
to load the intel-compilers-18 into the development environment:
module load intel-compilers-18
This will load the default version of the intel compilers. If you need a specific version of the module, you can add more information:
module load intel-compilers-18/18.0.5.274
will load version 18.0.2.274 for you, regardless of the default.
If a module loading file cannot be accessed within 10 seconds, a warning message will appear:
Warning: Module system not loaded
.
If you want to clean up, module remove
will remove a loaded module:
module remove intel-compilers-18
(or module rm intel-compilers-18
or
module unload intel-compilers-18
) will unload what ever version of
intel-compilers-18 (even if it is not the default) you might have
loaded. There are many situations in which you might want to change the
presently loaded version to a different one, such as trying the latest
version which is not yet the default or using a legacy version to keep
compatibility with old data. This can be achieved most easily by using
“module swap oldmodule newmodule”.
Suppose you have loaded version 18 of the Intel compilers; the following command will change to version 19:
module swap intel-compilers-18 intel-compilers-19
Available Compiler Suites¶
Note
As Cirrus uses dynamic linking by default you will generally also need to load any modules you used to compile your code in your job submission script when you run your code.
Intel Compiler Suite¶
The Intel compiler suite is accessed by loading the intel-compilers-*
and
intel-*/compilers
modules, where *
references the version. For example,
to load the 2019 release, you would run:
module load intel-compilers-19
Once you have loaded the module, the compilers are available as:
ifort
- Fortranicc
- Cicpc
- C++
See the extended section below for further details of available Intel compiler versions and tools.
GCC Compiler Suite¶
The GCC compiler suite is accessed by loading the gcc/*
modules, where
*
again is the version. For example, to load version 8.2.0 you would run:
module load gcc/8.2.0
Once you have loaded the module, the compilers are available as:
gfortran
- Fortrangcc
- Cg++
- C++
Compiling MPI codes¶
MPI on Cirrus is currently provided by the HPE MPT library.
You should also consult the chapter on running jobs through the batch system for examples of how to run jobs compiled against MPI.
Note
By default, all compilers produce dynamic executables on Cirrus. This means that you must load the same modules at runtime (usually in your job submission script) as you have loaded at compile time.
Using HPE MPT¶
To compile MPI code with HPE MPT, using any compiler, you must first load the “mpt” module.
module load mpt
This makes the compiler wrapper scripts mpicc
, mpicxx
and mpif90
available
to you.
What you do next depends on which compiler (Intel or GCC) you wish to use to compile your code.
Note
We recommend that you use the Intel compiler wherever possible to compile MPI applications as this is the method officially supported and tested by HPE.
Note
You can always check which compiler the MPI compiler wrapper scripts
are using with, for example, mpicc -v
or mpif90 -v
.
Using Intel Compilers and HPE MPT¶
Once you have loaded the MPT module you should next load the Intel compilers
module you intend to use (e.g. intel-compilers-19
):
module load intel-compilers-19
The compiler wrappers are then available as
mpif90
- Fortran with MPImpicc
- C with MPImpicxx
- C++ with MPI
Note
The MPT compiler wrappers use GCC by default rather than the Intel compilers:
When compiling C applications you must also specify that
mpicc
should use the icc
compiler with, for example,
mpicc -cc=icc
. Similarly, when compiling C++ applications
you must also specify that mpicxx
should use the icpc
compiler
with, for example, mpicxx -cxx=icpc
. (This is not required for
Fortran as the mpif90
compiler automatically uses ifort
.) If
in doubt use mpicc -cc=icc -v
or mpicxx -cxx=icpc -v
to see
which compiler is actually being called.
Alternatively, you can set the environment variables MPICC_CC=icc
and/or
MPICXX=icpc
to ensure the correct base compiler is used:
export MPICC_CC=icc
export MPICXX_CXX=icpc
Using GCC Compilers and HPE MPT¶
Once you have loaded the MPT module you should next load the
gcc
module:
module load gcc
Compilers are then available as
mpif90
- Fortran with MPImpicc
- C with MPImpicxx
- C++ with MPI
Note
HPE MPT does not support the syntax use mpi
in Fortran
applications with the GCC compiler gfortran
. You should use the
older include "mpif.h"
syntax when using GCC compilers with
mpif90
. If you cannot change this, then use the Intel compilers
with MPT.
Using Intel MPI¶
Although HPE MPT remains the default MPI library and we recommend
that first attempts at building code follow that route, you may
also choose to use Intel MPI if you wish. To use these, load the
appropriate intel-mpi
module, for example intel-mpi-19
:
module load intel-mpi-19
Please note that the name of the wrappers to use when compiling with Intel MPI depends on whether you are using the Intel compilers or GCC. You should make sure that you or any tools use the correct ones when building software.
Note
Although Intel MPI is available on Cirrus, HPE MPT remains the recommended and default MPI library to use when building applications.
Note
Using Intel MPI 18 can cause warnings in
your output similar to no hfi units are available
or
The /dev/hfi1_0 device failed to appear
. These warnings
can be safely ignored, or, if you would prefer to prevent
them, you may add the line
export I_MPI_FABRICS=shm:ofa
to your job scripts after loading the Intel MPI 18 module.
Note
When using Intel MPI 18, you should always launch MPI tasks
with srun
, the supported method on Cirrus. Launches
with mpirun
or mpiexec
will likely fail.
Using Intel Compilers and Intel MPI¶
After first loading Intel MPI, you should next load the appropriate
intel-compilers
module (e.g. intel-compilers-19
):
module load intel-compilers-19
You may then use the following MPI compiler wrappers:
mpiifort
- Fortran with MPImpiicc
- C with MPImpiicpc
- C++ with MPI
Using GCC Compilers and Intel MPI¶
After loading Intel MPI, you should next load the
gcc
module you wish to use:
module load gcc
You may then use these MPI compiler wrappers:
mpif90
- Fortran with MPImpicc
- C with MPImpicxx
- C++ with MPI
Compiler Information and Options¶
The manual pages for the different compiler suites are available:
- GCC
- Fortran
man gfortran
, C/C++man gcc
- Intel
- Fortran
man ifort
, C/C++man icc
Useful compiler options¶
Whilst difference codes will benefit from compiler optimisations in different ways, for reasonable performance on Cirrus, at least initially, we suggest the following compiler options:
- Intel
-O2
- GNU
-O2 -ftree-vectorize -funroll-loops -ffast-math
When you have a application that you are happy is working correctly and has reasonable performance you may wish to investigate some more aggressive compiler optimisations. Below is a list of some further optimisations that you can try on your application (Note: these optimisations may result in incorrect output for programs that depend on an exact implementation of IEEE or ISO rules/specifications for math functions):
- Intel
-fast
- GNU
-Ofast -funroll-loops
Vectorisation, which is one of the important compiler optimisations for Cirrus, is enabled by default as follows:
- Intel
- At
-O2
and above - GNU
- At
-O3
and above or when using-ftree-vectorize
To promote integer and real variables from four to eight byte precision for Fortran codes the following compiler flags can be used:
- Intel
-real-size 64 -integer-size 64 -xAVX
(Sometimes the Intel compiler incorrectly generates AVX2 instructions if the-real-size 64
or-r8
options are set. Using the-xAVX
option prevents this.)- GNU
-freal-4-real-8 -finteger-4-integer-8
Using static linking/libraries¶
By default, executables on Cirrus are built using shared/dynamic libraries (that is, libraries which are loaded at run-time as and when needed by the application) when using the wrapper scripts.
An application compiled this way to use shared/dynamic libraries will use the default version of the library installed on the system (just like any other Linux executable), even if the system modules were set differently at compile time. This means that the application may potentially be using slightly different object code each time the application runs as the defaults may change. This is usually the desired behaviour for many applications as any fixes or improvements to the default linked libraries are used without having to recompile the application, however some users may feel this is not the desired behaviour for their applications.
Alternatively, applications can be compiled to use static libraries (i.e. all of the object code of referenced libraries are contained in the executable file). This has the advantage that once an executable is created, whenever it is run in the future, it will always use the same object code (within the limit of changing runtime environment). However, executables compiled with static libraries have the potential disadvantage that when multiple instances are running simultaneously multiple copies of the libraries used are held in memory. This can lead to large amounts of memory being used to hold the executable and not application data.
To create an application that uses static libraries you must
pass an extra flag during compilation, -Bstatic
.
Use the UNIX command ldd exe_file
to check whether you are using an
executable that depends on shared libraries. This utility will also
report the shared libraries this executable will use if it has been
dynamically linked.
Intel modules and tools¶
There are a number of different Intel compiler versions available, and there is also a slight difference in the way different versions appear.
A full list is available via module avail intel
.
The different available compiler versions are:
intel-*/18.0.5.274
Intel 2018 Update 4intel-*/19.0.0.117
Intel 2019 Initial releaseintel-19.5/*
Intel 2019 Update 5intel-20.4/*
Intel 2020 Update 4
We recommend the most up-to-date version in the first instance, unless you have particular reasons for preferring an older version.
For a note on Intel compiler version numbers, see this Intel page
The different module names (or parts thereof) indicate:
cc
C/C++ compilers onlycmkl
MKL libraries (see Software Libraries section)compilers
Both C/C++ and Fortran compilersfc
Fortran compiler onlyitac
Intel Trace Analyze and Collectormpi
Intel MPIpxse
Intel Parallel Studio (all Intel modules)tbb
Thread Building Blocksvtune
VTune profiler - note that in older versions (intel-*/18.0.5.274
,intel-*/19.0.0.117
VTune is launched asamplxe-gui
for GUI oramplxe-cl
for CLI use)