setup wsl2 conda mamba and cuda
best of breed windows + linux
wsl2 - installation and configuration
most of it is explained in an internal blog entry.
To display windows version: winver.exe
I use version 20H2 build 19042.1415
installation
It is now as easy as to run wsl --install
in powershell as admin.
Full detail at MS WSL doc
Other commands:
# list all wsl distributions installed and their WSL version
wsl --list --verbose
# list all distributions available
wsl --list --online
Voici la liste des distributions valides qui peuvent être installées.
Installer à l’aide de « wsl --install -d <Distribution> ».
NAME FRIENDLY NAME
Ubuntu Ubuntu
Debian Debian GNU/Linux
kali-linux Kali Linux Rolling
openSUSE-42 openSUSE Leap 42
SLES-12 SUSE Linux Enterprise Server v12
Ubuntu-16.04 Ubuntu 16.04 LTS
Ubuntu-18.04 Ubuntu 18.04 LTS
Ubuntu-20.04 Ubuntu 20.04 LTS
# install Linux distributions
wsl --install -d <Distribution Name>
# shutdown wsl: shutdown all
wsl --shutdown
# define default wsl distribution to use with wsl
wsl -s <DistributionName>
installation in a non-system drive
and here is a more advanced config to install in a non system drive (D: instead of C:)
#from powershell
New-Item D:\WSL\Ubuntu-20.04 -ItemType Directory
Set-Location D:\WSL\Ubuntu-20.04
#list+link of distributions in https://docs.microsoft.com/en-us/windows/wsl/install-manual#downloading-distributions
Invoke-WebRequest -Uri https://aka.ms/wslubuntu2004 -OutFile Ubuntu-20.04.appx -UseBasicParsing
Rename-Item .\Ubuntu-20.04.appx Ubuntu-20.04.zip
Expand-Archive .\Ubuntu-20.04.zip -Verbose
# and then run Ubuntu_2004.2021.825.0_x64.appx
I don’t know yet where the WSL disk is located (is it a .vhdx file?)
Disks are located at %USERPROFILE%\AppData\Local\Packages\[distro name]
2 ditros used:
-
wsl1 ubuntu 18.04: CanonicalGroupLimited.Ubuntu18.04onWindows_79rhkp1fndgsc
-
wsl2 ubuntu 20.04: CanonicalGroupLimited.UbuntuonWindows_79rhkp1fndgsc
So disks are still in C:\
Guess I have to use move-wsl.
Set-Location 'D:\Program Files (x86)\move-wsl\'
.\move-wsl.ps1
PS D:\Program Files (x86)\move-wsl> .\move-wsl.ps1
Getting distros...
Select distro to move:
1: Ubuntu-18.04
2: Ubuntu
2
Enter WSL target directory:
D:\wsl\Ubuntu-20.04
Move Ubuntu to "D:\wsl\Ubuntu-20.04"? (Y|n): Y
Exporting VHDX to "D:\wsl\Ubuntu-20.04\Ubuntu.tar" ...
And after that, have to create file/etc/wsl.conf
guillaume@LL11LPC0PQARQ:~$ cat /etc/wsl.conf
[user]
default=guillaume
configuration
wsl-vpnkit
It is nicely explained in the Michelin blog entry. DNS resolution is kind of broken (I think due to internal protections we use on our corporate PC)
vpnkit provides a secured solution to make it work. And sakai135 has packaged it for wsl: wsl-vpnkit
The steps to install wsl-vpn kit are:
- Create a working directory on your windows workspace and download this packaging of wsl-vpnkit inside.
- Now, open a powershell and go to the location of wsl-vpnkit.tar.gz, downloaded during the previous step
- On your powershell terminal, launch:
#/!\ in powserhsell
wsl --import wsl-vpnkit $env:USERPROFILE\wsl-vpnkit wsl-vpnkit.tar.gz
wsl -d wsl-vpnkit
- You can now exit your powershell
- For the last step, to ensure all wsl reboot good communication, we will write in .profile file of your ubuntu user wsl-vpnkit initialization command:
echo 'wsl.exe -d wsl-vpnkit service wsl-vpnkit start' >> ~/.profile
- Relaunch your WSL terminal
git configuration
create a SSH key pair under your distribution
ssh-keygen -t rsa -b 4096 -C "WSL2"
Integrate into gitlab using gitlab doc. (copy id_rsa.pub
into gitlab > preferences > SSH Keys)cat .s
Add corporate CA certificates
Michelin SI is behind an ssl proxy with his proper PKI for certificates delivering. That is why, your subsystem must add this pki in her recognized authorities.
To do this, we will clone a repository with the certificates in the subsystem and copy them to ca-certificates:
git clone git@gitlab.michelin.com:devops-foundation/devops_environment.git /tmp/devops_environment
sudo cp /tmp/devops_environment/certs/* /usr/local/share/ca-certificates/
sudo update-ca-certificates
If everything is ok, terminal notify you that certificates has been added.
You can now clean the temp working folder:
rm -rf /tmp/devops_environment
we can have similar approach to update CA certifcates for Python. 1st step is to locate cacert.pem of your active python environment.
import certifi
certifi.where()
>> '/home/guillaume/miniconda3/envs/fastai/lib/python3.9/site-packages/certifi/cacert.pem'
TO BE FIXED
After having run update-ca-certifcates
, there is an updated ca file at /etc/ssl/certs/ca-certificates.crt
. Let’s concatenate it to our cacert.pem
.
cp /etc/ssl/certs/ca-certificates.crt /tmp/ca-certificates.crt
openssl x509 -in /tmp/ca-certificates.crt -out /tmp/ca-certificates.pem -outform PEM
cat /tmp/ca-certificates.pem | tee -a /home/guillaume/miniconda3/envs/fastai/lib/python3.9/site-packages/certifi/cacert.pem
Configure APT
The last step, to have a subsystem ready to use, is to have an apt with Michelin trusted sources configured. Ubuntu based package repositories can’t be used behind Michelin proxy.
Michelin offers its own apt server with artifactory. To configure apt to use artifactory, launch these commands:
echo 'Acquire { http::User-Agent "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:13.37) Gecko/20100101 Firefox/31.33.7"; };' | sudo tee /etc/apt/apt.conf.d/90globalprotectconf
sudo sed -i 's@^\(deb \)http://archive.ubuntu.com/ubuntu/\( focal\(-updates\)\?.*\)$@\1https://artifactory.michelin.com/artifactory/ubuntu-archive-remote\2\n# &@' /etc/apt/sources.list
sudo sed -i 's@^\(deb \)http://security.ubuntu.com/ubuntu/\( focal\(-updates\)\?.*\)$@\1https://artifactory.michelin.com/artifactory/ubuntu-security-remote\2\n# &@' /etc/apt/sources.list
Check if your WSL distribution is working correctly
To verify if everything is OK on your distribution:
- This command must return google ip:
host google.fr
- This command must return artifactory ip:
host artifactory.michelin.com
- You are able to update your distribution without error:
sudo apt update
sudo apt upgrade -y
Conda Mamba - installation and configuration
conda installation
tmpdir=$(mktemp -d)
cd $tmpdir
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
chmod +x Miniconda3-latest-Linux-x86_64.sh
# answer yes to question Do you wish the installer to initialize Miniconda3 by running conda init?
bash Miniconda3-latest-Linux-x86_64.sh -p $HOME/miniconda3
>> ==> For changes to take effect, close and re-open your current shell. <==
With this configuration, conda will be activate at startup. If you’d prefer that conda’s base environment not be activated on startup, set the auto_activate_base parameter to false: conda config --set auto_activate_base false
conda configuration
As we have in-house CA certificates, and conda uses its own CA certificates (in ~/miniconda3/ssl
)
We have to change this behaviour and ask conda to use system CA certifcates.
At the end of .bash_rc
, add export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
mamba installation
conda install mamba -n base -c conda-forge
mamba init
installation jupyter notebook, nb_conda_kernels, jupyter lab
mamba install nb_conda_kernels
mamba install -c conda-forge jupyterlab jupyterlab-git
create conda envt - fastai (v2.5.3) (optional)
mamba create --name fastai
mamba activate fastai
mamba install -c fastai -c pytorch fastai
mamba install ipykernel
Jupyter
Start jupyter (lab or notebook) from base environment, and switch to desired python environment.
modify jupyter config
#create jupyter config file in ~.jupyter
jupyter notebook --generate-config
And activate jupyter config file to change #c.NotebookApp.use_redirect_file = True
to c.NotebookApp.use_redirect_file = False