Introduction
When working on an shared HPC cluster (supercomputer), write permissions are generally limited to your ${HOME} directory or group shared/public dirs. Installing Python packages is generally not a problem, especially when using virtual environments, miniconda, etc. because these get installed in your home dir. However, sometimes it’s important to isolate an Python installation to a specific location that’s outside your home dir. This could be helpful when building reproducible pipelines or simple toolsets that other users on the system can access easily.
The main trick is to specify an environment variable, PYTHONUSERBASE, and set it to the base of your desired installation location.
Demo
For example, I will install GMM-demux (a tool for demultiplexing single-cell RNAseq hashtag sequences) from PyPi.
- First, set some variables for your software tool
# Make sure you have python available in your path
python --version
# Create some variables for the software names
MODULE_NAME="gmm_demux"
VERSION="0.2.2.1"
MODULES_DIR="/home/lmnp/shared/modules"- Then create a new
PYTHONUSERBASEvariable, and export - Make the directory if it doesn’t exist
# This is where the files will get installed
PYTHONUSERBASE="${MODULES_DIR}/${MODULE_NAME}/${VERSION}"
export PYTHONUSERBASE
# Set up a folder for the installation
mkdir -p $PYTHONUSERBASE
cd $PYTHONUSERBASE- To find available versions with
pip, try installing without a version number (e.g.pip install GMM_Demux==) - Install with the
--user --ignore-installedflags sopipwill act as if your specifiedPYTHONUSERBASEis your new home dir and install dependencies
pip install --user --ignore-installed GMM_Demux==${VERSION}- Finally, if you want to use the tool, make sure to update your path variables — and share with your installation with others!
export PATH="${MODULES_DIR}/${MODULE_NAME}/${VERSION}/bin:${PATH}"
export PYTHONPATH="${MODULES_DIR}/${MODULE_NAME}/${VERSION}/lib/python3.9/site-packages:${PYTHONPATH}"
# Test the help page
GMM_demux --help