Solid state storage¶
In addition to the Lustre file system, the Cirrus login and compute nodes have access to a shared, high-performance, solid state storage system (also known as RPOOL). This storage system is network mounted and shared across the login nodes and GPU compute nodes in a similar way to the normal, spinning-disk Lustre file system but has different performanc characteristics.
The solid state storage has a maximum usable capacity of 256 TB which is shared between all users.
Backups, quotas and data longevity¶
There are no backups of any data on the solid state storage so you should ensure that you have copies of critical data elsewhere.
In addition, the solid state storage does not currently have any quotas (user or group) enabled so all users are potentially able to access the full 256 TB capacity of the storage system. We ask all users to be considerate in their use of this shared storage system and to delete any data on the solid state storage as soon as it no longer needs to be there.
We monitor the usage of the storage system by users and groups and will potentially remove data that is stopping other users getting fair access to the storage and data that has not been actively used for long periods of time.
Accessing the solid-state storage¶
You access the solid-state storage at /scratch/space1 on both the login nodes and
on the compute nodes.
Everybody has access to be able to create directories and add data so we suggest
that you create a directory for your project and/or user to avoid clashes with files
and data added by other users. For example, if my project is t01 and my
username is auser then I could create a directory with
mkdir -p /scratch/space1/t01/auser
When these directories are initially created they will be world-readable. If you do not want users from other projects to be able to see your data, you should change the permissions on your new directory. For example, to restrict the directory so that only other users in your project can read the data you would use:
chmod -R o-rwx /scratch/space1/t01
Copying data to/from solid-state storage¶
You can move data to/from the solid-state storage in a number of different ways:
- By copying to/from another Cirrus file system - either interactively on login nodes or as part of a job submission script
- By transferring directly to/from an external host via the login nodes
Local data transfer¶
The most efficient tool for copying to/from the Cirrus file systems (/home, /work) to
the solid state storage is generally the cp command, e.g.
cp -r /path/to/data-dir /scratch/space1/t01/auser/
where /path/to/data-dir should be replaced with the path to the data directory you are
wanting to copy and assuming, of course, that you have setup the t01/auser subdirectories
as described above).
Note
If you are transferring data from your /work directory, these commands can also
be added to job submission scripts running on the compute nodes to move data as
part of the job. If you do this, remember to include the data transfer time in the
overall walltime for the job.
Data from your /home directory is not available from the compute nodes and must
therefore be transferred from a login node.
Remote data transfer¶
You can transfer data directly to the solid state storage from external locations using
scp or rsync in exactly the same way as you would usually do to transfer data
to Cirrus. Simply substitute the path to the location on the solid state storage for that
you would normally use for Cirrus. For example, if you are on the external location (e.g.
your laptop), you could use something like:
scp -r data_dir user@login.cirrus.ac.uk:/scratch/space1/t01/auser/
You can also use commands such as wget and curl to pull data from external
locations directly to the solid state storage.
Note
You cannot transfer data from external locations in job scripts as the Cirrus compute nodes do not have external network access.