Install Jupyter in a virtual environment

Create and setup the virtual environment:

python3 -m virtualenv env-name
source env-name/bin/activate
pip3 install jupyter

Run Jupyter Notebook:

jupyter notebook

While it is very easy to install modules by sourcing the correct virtualenv and using pip, there are a few drawbacks to this approach. Jupyter Notebook needs to be installed for each new environment created. If used as a server, this would become quite tedious and difficult to maintain - particularly if there is a need to use multiple environemnts at once with different users.

Install Jupyter as global system package

The Jupyter Notebook installation can be centralized using the system package manager. Virtual environments are created separately.

To install Jupyter Notebook on Debian based systems, use apt:

sudo apt install python3 python3-virtualenv jupyter
jupyter notebook

This is convenient because there is only one Jupyter Notebook installation - it can be accessed on localhost:8888. Next, virtual environments must created, and modules need to be installed in different virtual environments.

Commands

Commands can be run from within a notebook. A better idea of how Jupyter Notebook works is needed to understand this. Internally, it has a component called the IPython Kernel. This is responsible for communicating with the Python interpreter on the system. The ipykernel is included by default when Jupyter Notebook is installed, but it can also be installed independently.

Shell commands are provided by the operating system. They can be run by prefixing a line with an exclaimation mark:

!ls ; cd example_dir

This spawns a subprocess that exits upon completion; it is equivalent (on Linux) to:

import subprocess
subprocess.run(["bash", "-c", "ls ; cd example_dir"])

In this case, the ls will list the files in the current directory and cd will change the directory. The change in directory will not persist since the process exists after completion.

Magic commands (prefixed with a percentage sign) are provided by the IPython kernel. State changes persist because these commands are run inside the kernel process itself (and apply to the virtual environment in which the kernel was installed):

%cd example_dir
%page my_file.txt
%pip install bs4

This prints example_dir/my_file.txt and installs BeautifulSoup (a module for HTML parsing) in the current virtual environment.

For all available magic commands, refer to the IPython documentation or run:

%lsmagic

To install a module in the current virtual environment:

%pip install module_name

It is not possible to create and use a virtual environment directly within a notebook. For example:

!python -m virtualenv myvenv
!source myvenv/bin/activate
!pip install bs4

The first two commands succeed but the third fails with an externally-managed-environment error. This is because a new subprocess is created for each command - the sourced environment is not actually activated when pip is used. This can be resolved by chaining the command:

!python -m virtualenv myvenv; source myvenv/bin/activate; pip install bs4

BeautifulSoup now installs in the myvenv virtual environment, but the notebook does not use this environment. The virtual environment that the IPython kernel is running on is different to the one created - virtual environments cannot be nested!

Multiple Virtual Environments

To make a virtual environments available to Jupyter Notebook, the ipykernel module should be installed inside the virtual environment:

python3 -m virtualenv --system-site-packages env-name
source env-name/bin/activate
pip install ipykernel
python3 -m ipykernel install --user --name=env-name --display-name "Python (env-name)"

The --system-site-packages flag is particularly important if you depend on globally installed modules. For example, matplotlib has to be specially packaged for ARM platforms like Raspberry Pi and installed using the system package manager. This flag ensures that these modules can be found.

Since the Jupyter Notebook installation is global, the above commands can be easily run from the interface by creating a new terminal.

When creating a new notebook, you can select Python (env-name) to use a particular virtual environment.

To list all kernels and remove a particular kernel:

jupyter-kernelspec list
jupyter-kernelspec uninstall env-name

There is still a problem - this is running directly on the server. If something goes wrong, it might affect other services. This is also not very easily portable.

Install Jupyter on Docker

Using docker, this can be containerized!

This Dockerfile uses Alpine linux as the base image and installs python, pip, virtualenv and jupyter-notebook using the Alpine package manager apk. The last command starts the notebook. This probably isn’t very secure - you might want to modify the flags based on your security model.

FROM alpine:3.22
RUN apk add python3 py3-pip py3-virtualenv jupyter-notebook
EXPOSE 8888
CMD ["jupyter-notebook", "--ip=0.0.0.0", "--no-browser", "--allow-root", "--NotebookApp.token='your-very-long-and-secure-passphrase'"]

Now, creating the following docker-compose.yaml file and running docker compose up will build the image and expose Jupyter Notebook on localhost:4444. To manage the virtual environments, use the terminal and follow the steps in the previous section.

services:
  jupyter:
    container_name: jupyter
    build: .
    ports:
      - 4444:8888

To reproduce this cionfiguration on a different service, just the Dockerfile and docker-compose.yaml need to be copied.