By far the most common issue with getting started in the HPC environment are path issues. Your ‘PATH’ is an environment variable where locations for software, libraries and data can be set. Without setting up paths correctly, wrong versions of software may get used, or not located at all, resulting in job failures on the cluster.
Software on the cluster
Most software for cluster computing are in /workspace/software/bin which are symbolic links to various versions of common software located in /workspace/software/. You should always use R, MATLAB, Python, etc. from this directory as this software are specially configured for the HPC environment. You may also build/compile your own software if you choose and keep it in /workspace/[username] but you must be careful that it is compiled on one of the cluster nodes so that it runs properly.
Local software
Some software installed by default by Ubuntu 14.04, such as R, has been removed from /usr/bin. This is done to ensure that the properly built R is sourced by the user from /workspace/software instead.
Modules
In many cases there is a module for the specific software or task you’re going to use. To see a list of available modules type the following
[user@lunchbox] (39)$ module avail
-------- /workspace/software/modules-tcl-1.923/modulefiles -------------
course/stat479 dot module-git module-info null openmpi/mpi-3.0.0 python/python2.7.14 python/python3.6.4 R/R-3.4.2 R/R-3.4.3 R/R-3.4.3-mkl R/R-3.4.4 R/R-3.4.4-mkl use.own
To load a module use the module load command. Software is organized by software-name/version.
[user@lunchbox] (40)$ which R
[user@lunchbox] (7)$ module load R/R-3.5.2
[user@lunchbox] (8)$ which R
/workspace/software/R-3.5.2/bin/R
You can see that before we loaded the R/R-3.5.2 module, we had no path to any R. After loading the module, we now have the correct path to that version of R set. You should use the module command whenever possible in your submit scripts. See the Submitting Jobs section for some examples.
Creating modules
If you are using the HPC cluster for teaching purposes, we can create modules for your class upon request. A module can set paths, run other scripts, change environment variables, etc. in order to setup a student’s work environment appropriately. Sometimes you may not wish for a student to be burdened with linux things when you simply want to teach the statistical concept firstly. It is important, however, that some knowledge of the linux environment be mastered and to not completely rely on the module system. Understand what’s going on!
Conda and Python
Python is special because its many packages, versions, and dependencies can often create discrepancies between each other if not handled carefully. It’s best to use a package and environment manager such as Anaconda or Miniconda. These are provided for you on the cluster in /workspace/software However, instead of using python and packages directly from /workspace/software/anaconda3 it’s recommended to create your own virtual environment. First, choose between anaconda or miniconda. Anaconda includes all python packages. There are many, so using anaconda will be large and have lots of extra packages you won’t need, however, it will have all the common things you’re looking for such as numpy, scipy, scikit-learn, etc. Miniconda is essentially an empty environment and you tell it which packages to include.
Anaconda example
[user@lunchbox] (40)$ module load python/anaconda
[user@lunchbox] (41)$ conda create -p /workspace/[username]/myEnvironment anaconda
[user@lunchbox] (42)$ source activate /workspace/[username]/myEnvironment
[user@lunchbox] (43)$ python
The above example loads the anaconda module so that we have all our conda commands available in our path. We then create our environment with the conda create command and specify the path to our workspace directory and include the package anaconda at the end to instruct this environment to import everything within the anaconda distribution. Finally, we activate our environment so that we’ll use the python and packages in there as opposed to the ones on the current system (/usr/bin) by using source activate /workspace/[username]/myEnvironment. Note that ‘myEnvironment’ is just a name you create to name your environment directory.
Miniconda example
[user@lunchbox] (40)$ module load python/miniconda
[user@lunchbox] (41)$ conda create -p /workspace/[username]/myEnvironment python=3.4 torchvision scipy
[user@lunchbox] (42)$ source activate /workspace/[username]/myEnvironment
[user@lunchbox] (43)$ python
In this scenario we create our python environment by specifically requesting a python version and a specific package. Nothing else will be installed. We have asked for python version 3.4 and the package torchvision and scipy.
Please see the Tutorials and Examples page for how to use your conda environments when submitting to the cluster.