This document is under active development and discussion!

If you find errors or omissions in this document, please don’t hesitate to submit an issue or open a pull request with a fix. We also encourage you to ask questions and discuss any aspects of the project on the Feel++ Gitter forum. New contributors are always welcome!


The proper ressources cluster of the Research Institute Advanced Mathematics (IRMA - UMR7501).

1. Prerequisites

  • You have to own an account on the machine. External user can ask for an account if they are related to an institute research project.

  • To learn how to use the cluster, we recommend reading the official documentation

  • To use differents libraries version that match your need, the cluster uses environment modules You should familiarize first to know how to load specific softwares.

  • The cluster use slurm job supervisor. You should be familiar with job creation and job submission before going further.

We provide on the cluster Feel profile modules which loads all Feel dependencies and that are guaranty to works! you can type module avail 2>&1 | grep profile to list all profiles.

2. Atlas Feel++ configuration

The cluster mount points with different filesystems with more or less fast read/write operations.

Table 1. Atlas main mount points
Mountpoint Description Usage

/home/,

Slow I/O operations (NFS file system),

Network stored data (documents)

/data/,

Normal I/O operations (Hard Drive),

Store your computed data

/ssd/,

Fast I/O operations (Solid state drives),

Compute

Set the FEELPP_WORKDIR environment variable to the working directory path. By default, this variable point to

export FEELPP_WORKDIR=/ssd/${USER}/feel

If your application generate an high amount of data, you should use /data/${USER}/feel instead.

3. Feel++ usage

Work in progress

In this section, we will learn basic usage of Feel on the cluster. We distinguish two cases, the first one uses the existing Feel module. You might want to modify the Feel++ library itself, the second case will explain how to proceed.

There are three main methods to use Feel++ from the simplest to the hardest:

  1. Using containers: singularity (recommended), docker.

  2. Compiling a Feel++ application using Feel++ library installed on the system (Feel++ atlas module).

  3. Compiling Feel++ from scratch (library + applications).

But first, we have to configure the cluster

3.1. Feel++ containers

INFO: Difficulty : easy

Singularity is the recommended method for Feel++ containers. Please read singularity section before going further.

To use singularity on the cluster, load the specific module:

module load singularity/2.2.1

We provide currently one singularity image container in the directory /data/images.

Contrary to docker, singularity does not allow to modify a container for security purposes, root priviledges are required.

First of all, copy the image in your ${HOME} directory

cp /data/images/feelpp-toolboxes.img ${HOME}

To run the container, run the following command

singularity shell -c -B ${FEELPP_WORKDIR}/feel:/feel ${HOME}/feelpp-toolboxes.img

Now you are in the container, you should have the bash prompt looking like this:

singularity prompt1
Always use the -c option (equivalent to -B /home/user:/home/user) ? to avoid sharing your home directory within the container! Security reason, a container can perform some operation on home directory. Configurations files on the host /home/user directory can create conflicts and produce unwanted behaviour.
Modify the Feel++ singularity image

Root priviledge are required to modify an image. To proceed, you should first install singularity on your machine. On your machine, copy a Feel++ image locally

scp user@irma-atlas:/data/images/feelpp-toolboxes.img ${HOME}

Now you can modify the image in root enabling write access -w

sudo singularity shell -c -w ${HOME}/feelpp-toolboxes.img

See singularity documentation for more details.

Job slurm

An example of slurm batch script using singularity.

myjob.slurm
#! /bin/bash
#SBATCH -p public
#SBATCH --export=ALL
#SBATCH -n 8
#SBATCH -N 1
#SBATCH -t 00:30:00
#SBATCH -D /data/scratch/${USER}/slurm
#SBATCH -o slurm.%j.%N.out
#SBATCH -e slurm.%j.%N.err
#SBATCH -v

export FEELPP_SINGULARITY_IMAGE=${HOME}/feelpp-toolboxes.img
export FEELPP_WORKDIR_HOST=/ssd/${USER}/feel

source /etc/profile.d/modules.sh

module load singularity/2.2.1

mpirun --bind-to core \
    singularity exec \
    -B ${FEELPP_WORKDIR_HOST}:/feel \
    -c \
    ${FEELPP_SINGULARITY_IMAGE} \
    /usr/local/bin/feelpp_qs_laplacian_2d \
    --config-file=/opt/feelpp/Testcases/quickstart/laplacian/feelpp2d/feelpp2d.cfg

To send the job, just run the command

sbatch myjob.slurm

To view your job

squeue -u ${USER}

For more info about slurm, consult atlas wiki slurm or official slurm documentation.

Multi-node is not supported yet!

3.1.2. Docker

Due to security issues, only whitelisted users can have access to it. Please contact atlas admins if docker is required.

Refer to Feel++ docker section to see how to use docker.

3.2. Feel++ via modules

This way is recommended if you only need to use the Feel++ library without any modification. Currently, a Feel++ module is provided for a nightly build version.

  • Load the Feel++ library via the module.

module load feelpp/nightly

{feel++} is now available in the system

the module set the environement variable FEELPP_DIR for the path to the Feel++ installation echo ${FEELPP_DIR}.
  • Load a feel++ profile to have access to all Feel++ requirements for compiling your application.

module load feelpp-clang_gcc610.profile
The profile should be the same used for the feel++ module to avoid mismatch setup for the compilation
From here you can go to Quickstart section.
  • Place yourself in a directory and create a cmake file named CMakeLists.txt containing the following code.

cmake_minimum_required(VERSION 2.8)

find_package(Feel++
  PATHS $ENV{FEELPP_DIR}/share/feel/cmake/modules
  /usr/share/feel/cmake/modules
  /usr/local/share/feel/cmake/modules
  /opt/share/feel/cmake/modules
  )
if(NOT FEELPP_FOUND)
  message(FATAL_ERROR "Feel++ was not found on your system. Make sure to install it and specify the FEELPP_DIR to reference the installation directory.")
endif()

feelpp_add_application(youApplication SRCS yourCode.cpp)

This file describes to cmake how to find the Feel++ library in the given directories. By default, cmake will search in the system default path.

  1. Create a C++ file named yourCode.cpp where you will write your first Feel++ code.

  2. Generate the Makefiles with cmake or using the configure script. /path/to/feelpp-sources/configure

  3. Compile you Feel++ application make