Quick Fix: How To Install Packages from the Jupyter Notebook

# Install a conda package in the current Jupyter kernel
import sys
!conda install --yes --prefix {sys.prefix} matplotlib

Collecting package metadata (current_repodata.json): done
Solving environment: done

# All requested packages already installed.

The Details: Why is Installation from Jupyter so Messy?

How your operating system locates executables

!echo $PATH

/home/explore/gems/bin:/home/explore/miniconda3/envs/pytorch/bin:/home/explore/miniconda3/condabin:/home/explore/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin

!type python

python is /home/explore/miniconda3/envs/pytorch/bin/python

You can optionally add the -a tag to see all available versions of the command in your current shell environment; for example:

!type -a python

python is /home/explore/miniconda3/envs/pytorch/bin/python
python is /usr/bin/python

!type -a conda

conda is /home/explore/miniconda3/condabin/conda

How Python locates packages

import sys
sys.path

['/home/explore/git/guillaume/blog/_notebooks',
 '/home/explore/miniconda3/envs/pytorch/lib/python38.zip',
 '/home/explore/miniconda3/envs/pytorch/lib/python3.8',
 '/home/explore/miniconda3/envs/pytorch/lib/python3.8/lib-dynload',
 '',
 '/home/explore/miniconda3/envs/pytorch/lib/python3.8/site-packages',
 '/home/explore/miniconda3/envs/pytorch/lib/python3.8/site-packages/IPython/extensions',
 '/home/explore/.ipython']

By default, the first place Python looks for a module is an empty path, meaning the current working directory. If the module is not found there, it goes down the list of locations until the module is found. You can find out which location has been used using the __path__ attribute of an imported module:

import numpy
numpy.__path__

['/home/explore/miniconda3/envs/pytorch/lib/python3.8/site-packages/numpy']

by printing the sys.path variables for each of the available python executables in my path, using Jupyter’s delightful ability to mix Python and bash commands in a single code block:

paths = !type -a python
for path in set(paths):
    path = path.split()[-1]
    print(path)
    !{path} -c "import sys; print(sys.path)"
    print()

/usr/bin/python
['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/home/explore/.local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/dist-packages', '/usr/local/lib/python2.7/dist-packages/PyCapture2-0.0.0-py2.7-linux-x86_64.egg', '/usr/lib/python2.7/dist-packages']

/home/explore/miniconda3/envs/pytorch/bin/python
['', '/home/explore/miniconda3/envs/pytorch/lib/python38.zip', '/home/explore/miniconda3/envs/pytorch/lib/python3.8', '/home/explore/miniconda3/envs/pytorch/lib/python3.8/lib-dynload', '/home/explore/miniconda3/envs/pytorch/lib/python3.8/site-packages']

pip install will install in the Python in the same path:

!type pip

pip is /home/explore/miniconda3/envs/pytorch/bin/pip

conda install will install in the active conda envt

!conda env list

# conda environments:
#
base                     /home/explore/miniconda3
d059                     /home/explore/miniconda3/envs/d059
datacamp                 /home/explore/miniconda3/envs/datacamp
deeplearning_specialization     /home/explore/miniconda3/envs/deeplearning_specialization
deeplearning_specialization_keras     /home/explore/miniconda3/envs/deeplearning_specialization_keras
deeplearning_specialization_tf1     /home/explore/miniconda3/envs/deeplearning_specialization_tf1
drl_handson              /home/explore/miniconda3/envs/drl_handson
fastai                   /home/explore/miniconda3/envs/fastai
gan                      /home/explore/miniconda3/envs/gan
gan_tensorflow           /home/explore/miniconda3/envs/gan_tensorflow
mit_6002x                /home/explore/miniconda3/envs/mit_6002x
pytorch               *  /home/explore/miniconda3/envs/pytorch
squeezebox               /home/explore/miniconda3/envs/squeezebox

The reason both pip and conda default to the conda pytorch environment is that this is the Python environment I used to launch the notebook.

How Jupyter executes code: Jupyter Kernels

!jupyter kernelspec list

Available kernels:
  python2    /home/explore/.local/share/jupyter/kernels/python2
  python3    /home/explore/miniconda3/envs/pytorch/share/jupyter/kernels/python3

!cat /home/explore/miniconda3/envs/pytorch/share/jupyter/kernels/python3/kernel.json

{
 "argv": [
  "/home/explore/miniconda3/envs/pytorch/bin/python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python 3",
 "language": "python"
}

If you’d like to create a new kernel, you can do so using the jupyter ipykernel command; for example, I created the above kernels for my primary conda environments using the following as a template:

$ source activate myenv
$ python -m ipykernel install --user --name myenv --display-name "Python (myenv)"

The Root of the Issue

The root of the issue is this: the shell environment is determined when the Jupyter notebook is launched, while the Python executable is determined by the kernel, and the two do not necessarily match. In other words, there is no guarantee that the python, pip, and conda in your $PATH will be compatible with the python executable used by the notebook.

Recall that the python in your path can be determined using

!type python

python is /home/explore/miniconda3/envs/pytorch/bin/python

The Python executable being used in the notebook can be determined using

sys.executable

'/home/explore/miniconda3/envs/pytorch/bin/python'

Note

when the 2 differs, boom!