Keeping track of python environments

I manage all my python environments with conda from miniconda.

manual way

However I don’t have a strong process to keep track of my environment specifications. Usually I manually create an env.txt file under my projects. Keeping all commands I have used to create that environment.

!cat /home/explore/git/guillaume/mit_6S191_Intro_to_deep_learning/env\ \ mit_6S191.txt

env_name: mit_6S191
libraries: python 3.7, tensorflow 2


Installation commands:
conda create -n mit_6S191 python=3.7
conda activate mit_6S191

conda install tensorflow tensorflow-gpu
conda install -c conda-forge jupyter_contrib_nbextensions
conda install matplotlib numpy opencv
conda install -c pytorch torchvision
conda install nb_conda

What happens if I add packages in that environment. Or want to use that environment in another project. I have to remember the link between env name and project name.

That is not robust.

yml way

Keeping a yml file could be a solution to keep track of environment specifications. It doesn’t answer to my last concern though (linking env name and project name)

But there is a limitation linked with channels.

!conda env export --from-history

name: fastai
channels:
  - defaults
dependencies:
  - python=3.8
  - fastai
  - jupyter
  - jupyter_contrib_nbextensions
  - fastbook
prefix: /home/explore/miniconda3/envs/fastai

In that example, fastai package should come from fastai channel but conda doesn’t keep that information.

Using

conda install -n my_env rdkit::rdkit

could be an option.

automate yml way

Since conda keeps active environment in env variable CONDA_DEFAULT_ENV, we can automatically create up-to-date yml file.

!echo $CONDA_DEFAULT_ENV

fastai

!conda env export --from-history > ~/temp/env_`echo $CONDA_DEFAULT_ENV`.yml
!ls ~/temp/env_`echo $CONDA_DEFAULT_ENV`.yml

/home/explore/temp/env_fastai.yml

But for it to be usable, I will have to install package using the <channel>::<package> way.

!cat /home/explore/git/guillaume/mit_6S191_Intro_to_deep_learning/create_yml.sh

#!/bin/bash
conda env export --from-history > env_`echo $CONDA_DEFAULT_ENV`.yml

Conda commands

When managing conda environments, I very often fall on this documentation page which is simply great: https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html

Next time I visit this page, I will enter entries here to track my common commands.

Jupyter installation

Jupyter extensions

I have already explained how to install jupyter extensions and the one I use. update jupyter to include extensions

nb_conda

This is usefull to switch from environment to another without having to stop/restart jupyter.