This tutorial covers releasing a Python distribution locally using GateWay as an example. It focuses on how to package a simple project in Python. It will show you how to add the necessary files and structure to create a package, how to build a package, and how to install it on any platform.
1. Preparatory activities
Please note that these steps are not mandatory to proceed successfully, however they are considered good practice.
1.1. PIP upgrade
Before any Python package activity, it is always a good move to upgrade your main Python package manager, such as PIP. This can be done from any location, however, sometimes the user is unable to do so due to file system permissions. In this case, it can be done for a local user using the --user option on the command line.
This tutorial is based on using GateWay, therefore a Python module shall be loaded first.
1.1.1. With necessary permissions
$ module load itm-python
$ python3 -m pip install --upgrade pip
1.1.2. Without necessary permissions
This solution is not recommended, but may be needed under certain circumstances, such as when the user does not have permissions.
$ module load itm-python
$ python3 -m pip install --upgrade pip --user
In that case it may be needed to add a local site-packages directory to $PATH:
$ PATH=$HOME/.local/bin:$PATH
1.2. Virtual Environment
At its core, the main purpose of Python virtual environments is to create an isolated environment for Python projects. This means that each project can have its own dependencies, regardless of what dependencies every other project has. The great advantage is that there is no limit to the number of environments that can be created as it is just a directory with a few scripts that handle system package isolation.
As shown in the previous section, there are at least two different locations where Python packages can be installed on the system. This can pose risks such as name shadowing or even suppression of certain packages from the kernel to the system. Most of the time, nothing like this happens, but even so, using a Python virtual environment is also considered good practice. This tutorial assumes usage of at least Python 3.6, therefore it should already have the venv module from the standard library installed.
1.2.1. Creating a Virtual Python Environment
# Python 3
$ python3 -m venv env_name
And that's just it. It is important to use the venv module as others are deprecated. The name of the environment can be anything as it is just a directory name, however it is good practice to use names like env or venv or expand with these names.
1.2.1.1. Create a Virtual Environment With System Site Package Inheritance
For packages developed on a clustered system such as GateWay, it may be necessary to inherit packages from the system site environment typically set with modules. To achieve this, add the --system-site-packages option when creating a new virtual environment.
$ python -m venv env_name --system-site-packages
1.2.2. Activate The Virtual Environment
To activate the environment just created, navigate to the location where it was created, if not already there, and execute the following command:
$ source env_name/bin/activate
(env_name): $
As a result of activation, the currently used shell should explicitly show the name of the environment being activated on the command line.
1.2.3. Deactivate The Virtual Environment
After finishing work with the environment, it can be deactivated in the following way:
(env_name): $ deactivate
Removing The Virtual Environment
Normally it is not necessary to remove the environment, but this can be achieved by simply deleting the env_name directory.
1.2.4. Upgrade PIP Inside The Virtual Environment
It is always good practice to update the PIP, even if it is up-to-date on the system site. This time the user should be able to access all files in the project directory, so there is no need to use the local --user flag with the install command. On GateWay, it should look like this:
# Python 3
$ python -m venv env_name --system-site-packages
$ source env_name/bin/activate
(env_name): $ python3 -m pip install --upgrade pip
2. Package directory starter
A the beginning the minimum starter project structure should look like below:
project_name
├── LICENSE
├── project_name
│ ├── __init__.py
│ └── project_name.py
├── README.md
├── requirements.txt
├── pyproject.toml
├── setup.cfg
├── setup.py
└── tests