User Conda Environments
Support for User Conda Environments
Please be aware that whilst users can create and utilise Conda environments on Baskerville Portal it is an unsupported feature.
Baskerville Portal’s JupyterLab app can find and utilise user-defined Conda environments. This is done by parsing the file ~/.conda/environments.txt and for each line, generating a kernel.
Creation process¶
You must first create a Conda environment. Due to the potential size of Conda environments, please ensure that you create the environment in an appropriate project directory and not in your home directory, which has a maximum quota of 20 GB.
Environment naming
Please ensure that your Conda environments have unique names.
Our recommendation is that you use Jupyter’s Launcher to access a Terminal on a compute node and then use the following boilerplate:
module purge; module load baskerville
module load bask-apps/live
module load Miniconda3/4.10.3
eval "$(${EBROOTMINICONDA3}/bin/conda shell.bash hook)"
# Define the path to your Conda environment (modify as appropriate)
# N.B. this path will be created by the subsequent commands if it doesn't already exist
CONDA_ENV_PATH="/bask/projects/_initial_/_projectname_/${USER}_conda_env"
# Create the environment. Only required once.
conda create --yes --prefix "${CONDA_ENV_PATH}"
# Activate the environment
conda activate "${CONDA_ENV_PATH}"
# Choose your version of Python
conda install --yes python=3.10
# Continue to install any further items as required.
# For example:
conda install --yes numpy
Where _initial_ is the first letter of the project name.
Note
When the app locates a compatible Conda environment it will check whether it has the necessary kernel module (e.g. ipykernel) and install it if required.
Usage and ongoing modifications¶
After initially creating your new Conda environment you will need to start a fresh JupyterLab app in Baskerville Portal so that it is made available as a kernel. However, once this is completed you can run a Notebook using the Conda kernel whilst simultaneously modifying the Conda environment using a complementary Terminal tab (see below).
You will need to start a new Terminal and run the following code each time you want to modify the environment:
module purge; module load baskerville
module load bask-apps/live
module load Miniconda3/4.10.3
eval "$(${EBROOTMINICONDA3}/bin/conda shell.bash hook)"
# Define the path to your existing Conda environment (modify as appropriate)
CONDA_ENV_PATH="/bask/projects/_initial_/_projectname_/${USER}_conda_env"
# Activate the environment
conda activate "${CONDA_ENV_PATH}"
# Make any modifications as required.
# For example:
conda install --yes requests
Note that there is no need to restart a running Conda kernel after making modifications – any changes will be available immediately.
modules and Conda environments
Do not load modules in combination with a Conda environment as you may experience issues. If you would like to switch from using modules to running User Conda Environments, and vice versa, then please shut down your JupyterLab session and relaunch a new interactive batch job.
Conda cache¶
Despite creating the Conda environment in a project directory (as covered above), by default Conda will still cache data in your home directory, which can rapidly increase in size and use-up the limited 20 GB quota. We therefore recommend that you perform one of the following options:
-
Periodically clean the cache by executing the following command in a Terminal window:
conda clean --all -
Prior to executing any Conda installation commands, export the following environment variable to a path with more storage space:
export CONDA_PKGS_DIRS=/tmp(See the storage documentation for further information on temporary storage on Baskerville.)