This document is under active development and discussion!

If you find errors or omissions in this document, please don’t hesitate to submit an issue or open a pull request with a fix. We also encourage you to ask questions and discuss any aspects of the project on the Feel++ Gitter forum. New contributors are always welcome!

The proper ressources cluster of the Research Institute Advanced Mathematics (IRMA - UMR7501).

1. Prerequisites

  • You have to own an account on the machine. External user can ask for an account if they are related to an institute research project.

  • To learn how to use the cluster, we recommend reading the official documentation

  • To use differents libraries version that match your need, the cluster uses environment modules You should familiarize first to know how to load specific softwares.

  • The cluster use slurm job supervisor. You should be familiar with job creation and job submission before going further.

We provide on the cluster Feel profile modules which loads all Feel dependencies and that are guaranty to works! you can type module avail 2>&1 | grep profile to list all profiles.

2. Atlas Feel++ configuration

The cluster mount points with different filesystems with more or less fast read/write operations.

Table 1. Atlas main mount points
Mountpoint Description Usage


Slow I/O operations (NFS file system),

Network stored data (documents)


Normal I/O operations (Hard Drive),

Store your computed data


Fast I/O operations (Solid state drives),


Set the FEELPP_WORKDIR environment variable to the working directory path. By default, this variable point to

export FEELPP_WORKDIR=/ssd/${USER}/feel

If your application generate an high amount of data, you should use /data/${USER}/feel instead.

3. Feel++ usage

Work in progress

In this section, we will learn basic usage of Feel on the cluster. We distinguish two cases, the first one uses the existing Feel module. You might want to modify the Feel++ library itself, the second case will explain how to proceed.

There are three main methods to use Feel++ from the simplest to the hardest:

  1. Using containers: singularity (recommended), docker.

  2. Compiling a Feel++ application using Feel++ library installed on the system (Feel++ atlas module).

  3. Compiling Feel++ from scratch (library + applications).

But first, we have to configure the cluster

3.1. Feel++ containers

INFO: Difficulty : easy

Singularity is the recommended method for Feel++ containers. Please read singularity section before going further.

To use singularity on the cluster, load the specific module:

module load singularity/2.2.1

We provide currently one singularity image container in the directory /data/images.

Contrary to docker, singularity does not allow to modify a container for security purposes, root priviledges are required.

First of all, copy the image in your ${HOME} directory

cp /data/images/feelpp-toolboxes.img ${HOME}

To run the container, run the following command

singularity shell -c -B ${FEELPP_WORKDIR}/feel:/feel ${HOME}/feelpp-toolboxes.img

Now you are in the container, you should have the bash prompt looking like this:

singularity prompt1
Always use the -c option (equivalent to -B /home/user:/home/user) ? to avoid sharing your home directory within the container! Security reason, a container can perform some operation on home directory. Configurations files on the host /home/user directory can create conflicts and produce unwanted behaviour.
Modify the Feel++ singularity image

Root priviledge are required to modify an image. To proceed, you should first install singularity on your machine. On your machine, copy a Feel++ image locally

scp user@irma-atlas:/data/images/feelpp-toolboxes.img ${HOME}

Now you can modify the image in root enabling write access -w

sudo singularity shell -c -w ${HOME}/feelpp-toolboxes.img

See singularity documentation for more details.

Job slurm

An example of slurm batch script using singularity.

#! /bin/bash
#SBATCH -p public
#SBATCH --export=ALL
#SBATCH -n 8
#SBATCH -t 00:30:00
#SBATCH -D /data/scratch/${USER}/slurm
#SBATCH -o slurm.%j.%N.out
#SBATCH -e slurm.%j.%N.err

export FEELPP_SINGULARITY_IMAGE=${HOME}/feelpp-toolboxes.img
export FEELPP_WORKDIR_HOST=/ssd/${USER}/feel

source /etc/profile.d/

module load singularity/2.2.1

mpirun --bind-to core \
    singularity exec \
    -B ${FEELPP_WORKDIR_HOST}:/feel \
    -c \
    /usr/local/bin/feelpp_qs_laplacian_2d \

To send the job, just run the command

sbatch myjob.slurm

To view your job

squeue -u ${USER}

For more info about slurm, consult atlas wiki slurm or official slurm documentation.

Multi-node is not supported yet!

3.1.2. Docker

Due to security issues, only whitelisted users can have access to it. Please contact atlas admins if docker is required.

Refer to Feel++ docker section to see how to use docker.

3.2. Feel++ via modules

This way is recommended if you only need to use the Feel++ library without any modification. Currently, a Feel++ module is provided for a nightly build version.

  • Load the Feel++ library via the module.

module load feelpp/nightly

{feel++} is now available in the system

the module set the environement variable FEELPP_DIR for the path to the Feel++ installation echo ${FEELPP_DIR}.
  • Load a feel++ profile to have access to all Feel++ requirements for compiling your application.

module load feelpp-clang_gcc610.profile
The profile should be the same used for the feel++ module to avoid mismatch setup for the compilation
From here you can go to Quickstart section.
  • Place yourself in a directory and create a cmake file named CMakeLists.txt containing the following code.

cmake_minimum_required(VERSION 2.8)

  PATHS $ENV{FEELPP_DIR}/share/feel/cmake/modules
  message(FATAL_ERROR "Feel++ was not found on your system. Make sure to install it and specify the FEELPP_DIR to reference the installation directory.")

feelpp_add_application(youApplication SRCS yourCode.cpp)

This file describes to cmake how to find the Feel++ library in the given directories. By default, cmake will search in the system default path.

  1. Create a C++ file named yourCode.cpp where you will write your first Feel++ code.

  2. Generate the Makefiles with cmake or using the configure script. /path/to/feelpp-sources/configure

  3. Compile you Feel++ application make

4. Post-processing

4.1. Paraview

4.1.1. Downloading the data

You can retrieve your data on you local machine using rsync.


and open the .case file

4.1.2. Distant connection (pvserver)

Open a terminal and connect to atlas server. The paraview Run the paraview server

module load paraview/5.1.0
use pvserver --multi-clients to connect with several users at the same time! See pvserver --help for all options.

On you machine, run paraview and connect to the server [file]→[connect]. A should pop out. Configure the server with the given address displayed in the terminal where you run pvserver. It should looks like cs://irma-atlas:11111.

Paraview and pvserver must be the same version!

4.2. Containers

4.2.1. Docker containers

Feel++ provides paraview directly in the docker image. First, you have to run a new container on atlas. Don’t forget to mount the volume /feel for Feel++ applications. We have to add an option to use the same network than irma-atlas --network=host

docker run --rm -it --network=host -v ${HOME}/feel:/feel feelpp/feelpp-toolboxes:develop-ubuntu-16.10

Then you can proceed with paraview server pvserver like in Distant connection (pvserver) section.

Distant connection (In-situ)

Feel++ is compatible with insitu thanks to kitware catalyst. First be sure to follow previous steps Docker containers to run a docker container.

In the container, start a paraview server in the background.

pvserver &

Keep in mind the printed address.

On you machine, run paraview and connect to this server like explained in section Distant connection (pvserver). Connect to catalyst via paraview menu [catalyst]→[connect]. A prompt message should inform you that paraview accept catalyst connections. Set catalyst to pause [catalyst]→[pause] in the menu before launching a Feel++ application.

Now in the container, chose an application you wish to execute. For example, let’s take ~/Testcases/FSI/wavepressure2d Run your application with insitu options.

mpirun -np 8 feelpp_toolbox_fsi_2d --config-file wavepressure2d.cfg --exporter.format=vtk --exporter.vtk.insitu.enable=1

The simulation will stop after the initialisation step waiting for you to resume catalyst.

Click on the icon to expand the object, then click on the "eye" icon to make the object visible.

At this time, you can add all filters as usual for post-processing.

Once you’re ready, resume catalyst [catalyst]→[continue]