SSL: CERTIFICATE_VERIFY_FAILED

have to rely on corporate certificate server
jupyter
Published

September 6, 2022

Description of the problem

From time to time when using network function, I have this kind of errors:

Traceback (most recent call last):
  File "/home/guillaume/miniconda/envs/dataset_tools/lib/python3.9/urllib/request.py", line 1346, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/home/guillaume/miniconda/envs/dataset_tools/lib/python3.9/http/client.py", line 1285, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/guillaume/miniconda/envs/dataset_tools/lib/python3.9/http/client.py", line 1331, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/home/guillaume/miniconda/envs/dataset_tools/lib/python3.9/http/client.py", line 1280, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/guillaume/miniconda/envs/dataset_tools/lib/python3.9/http/client.py", line 1040, in _send_output
    self.send(msg)
  File "/home/guillaume/miniconda/envs/dataset_tools/lib/python3.9/http/client.py", line 980, in send
    self.connect()
  File "/home/guillaume/miniconda/envs/dataset_tools/lib/python3.9/http/client.py", line 1454, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/home/guillaume/miniconda/envs/dataset_tools/lib/python3.9/ssl.py", line 501, in wrap_socket
    return self.sslsocket_class._create(
  File "/home/guillaume/miniconda/envs/dataset_tools/lib/python3.9/ssl.py", line 1041, in _create
    self.do_handshake()
  File "/home/guillaume/miniconda/envs/dataset_tools/lib/python3.9/ssl.py", line 1310, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)

Context

My company uses some ssl interceptor and it has to be considered as a cert autority.

Solution

from https://stackoverflow.com/questions/51390968/python-ssl-certificate-verify-error

where certifcates are kept

import certifi
pem_path = certifi.where() 
pem_path
'/home/guillaume/miniconda/envs/dataset_tools/lib/python3.9/site-packages/certifi/cacert.pem'

get company certificates

tmpdir = !mktemp -d
tmpdir
['/tmp/tmp.0V7L0xu2a5']
!git clone git@gitlab.michelin.com:DEV/bib-certificates.git {tmpdir[0]}
Cloning into '/tmp/tmp.0V7L0xu2a5'...
remote: Enumerating objects: 87, done.
remote: Total 87 (delta 0), reused 0 (delta 0), pack-reused 87
Receiving objects: 100% (87/87), 78.90 KiB | 1.55 MiB/s, done.
Resolving deltas: 100% (26/26), done.
!ls -l {tmpdir[0]}/*trust-ca.pem
-rw-r--r-- 1 guillaume guillaume 1606 Sep  6 19:11 /tmp/tmp.0V7L0xu2a5/cert_M_X5C_aze-cn-sslfwd-trust-ca.pem
-rw-r--r-- 1 guillaume guillaume 1606 Sep  6 19:11 /tmp/tmp.0V7L0xu2a5/cert_M_X5C_rnh-ac-sslfwd-trust-ca.pem
-rw-r--r-- 1 guillaume guillaume 1606 Sep  6 19:11 /tmp/tmp.0V7L0xu2a5/cert_M_X5C_rnh-eu-sslfwd-trust-ca.pem
-rw-r--r-- 1 guillaume guillaume 1606 Sep  6 19:11 /tmp/tmp.0V7L0xu2a5/cert_M_X5C_rnh-na-sslfwd-trust-ca.pem
-rw-r--r-- 1 guillaume guillaume 1602 Sep  6 19:11 /tmp/tmp.0V7L0xu2a5/cert_M_X5C_sase-mob-sslfwd-trust-ca.pem
-rw-r--r-- 1 guillaume guillaume 1602 Sep  6 19:11 /tmp/tmp.0V7L0xu2a5/cert_M_X5C_sase-net-sslfwd-trust-ca.pem
import os

for filename in os.listdir(tmpdir[0]):
    if filename.endswith("trust-ca.pem"): 
         # print(os.path.join(directory, filename))
        !cat {os.path.join(tmpdir[0], filename)} >> {pem_path}
        continue
    else:
        continue

Validate it works

import urllib.request
with urllib.request.urlopen('http://python.org/', cafile=certifi.where()) as response:
   html = response.read()
/tmp/ipykernel_2003/2808005746.py:2: DeprecationWarning: cafile, capath and cadefault are deprecated, use a custom context instead.
  with urllib.request.urlopen('http://python.org/', cafile=certifi.where()) as response:
html[:100]
b'<!doctype html>\n<!--[if lt IE 7]>   <html class="no-js ie6 lt-ie7 lt-ie8 lt-ie9">   <![endif]-->\n<!-'

and from a command-line

export SSL_CERT_FILE='/home/guillaume/miniconda/envs/dataset_tools/lib/python3.9/site-packages/certifi/cacert.pem'
nbdev_new

Integration in WSL2

I will modify SSL cert of my (base) environment.

and add export SSL_CERT_FILE in .bashrc

I have made the modification at install ubuntu 22.04 on WSL

!cat ../files/setup_wsl_02_install_python_conda_part3.sh
echo "configure SSL cert v2"

conda deactivate
pip install -U certifi
export SSL_CERT_FILE=`python -c 'import certifi;print(certifi.where())'`

export TMPDIR=`mktemp -d`
git clone git@gitlab.michelin.com:DEV/bib-certificates.git $TMPDIR
cd $TMPDIR
cat *trust-ca.pem >> $SSL_CERT_FILE

tee -a ~/.bashrc << EOF
export SSL_CERT_FILE=$SSL_CERT_FILE
EOF

if [ -e "/.cfg" ]; then
        config='/usr/bin/git --git-dir=/.cfg/ --work-tree=/'
        $config add ~/.bashrc
        $config commit -m'export certificates for commandline'
        $config push        
fi

To be (re)executed after certifi update

Of course each time certifi is being updated, we have to re-run this process.

Here is a script that could be run each time we have the same problem:

!cat /home/guillaume/_conda_env/update_SSL.sh
#!/bin/bash
conda deactivate
pip list|grep certifi
pip install -U certifi
pip list|grep certifi
export SSL_CERT_FILE=`python -c 'import certifi;print(certifi.where())'`

export TMPDIR=`mktemp -d`
git clone git@gitlab.michelin.com:DEV/bib-certificates.git $TMPDIR
cd $TMPDIR
cat *trust-ca.pem >> $SSL_CERT_FILE