Solving the Dreadful Certificate Issues in Python Requests Module

Recently I have been working with the Python requests module to secure an API call using the server’s certificate.

I was stuck at a point and learning what I did to fix that issue was great and led me to create a post ondeep dive with SSL certificates.

The Problem

I was using the requests module and here is the API call.


response = requests.post(url, files=files, headers=headers)

This is the error I was getting in return:

/ (Caused by SSLError(SSLCertVerification(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)'))

Attempts

Doing unsecured calls with verify=false

My first try was to use the verify flag as False and try.

response = requests.post(url, files=files, headers=headers, verify=False)

Though I got a 200, I got a nasty warning confirming that I am doing a horrible job, not providing a certificate. So I had to find the right way to do it.

Provide the server certificate

First I thought, if I can provide the server certificate in the verify key, it would do the trick. So I did,

response = requests.post(url, files=files, headers=headers, verify='server.cer')

This is a DER encoded certificate

I got another error:

/ (Caused by SSLError(SSLError(136, '[X509] no certificate or crl found (_ssl.c:4232)'))

Override CA_REQUESTS_BUNDLE

The module requests to use certifi to access the CA bundle and validate secure SSL connections and we can use the CA_REQUESTS_BUNDLE environment variable to override the CA bundle location. So I thought, if I can manually provide the server.cer in that variable, I will achieve enlightenment. But to my despair, that too failed.

Convert to different certificate encoding

Then I thought that the requests module must not be taking the DER encoded certificate. So I converted to PEM which is plaintext but Base 64 encoded.

openssl x509 -in server.cer -inform DER -outform PEM -out server.pem

and did the call again

response = requests.post(url, files=files, headers=headers, verify='server.pem')

Another error.

/ (Caused by SSLError(SSLCertVerification(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)'))

Yes, this is the same as Error 1. So we are back to square 1. I tried to search the internet, but no one had a meaningful solution. Someone

Then I thought this is not the way, I will solve this issue. So I did a lot of studying for the Certificates and finally, I got them. I wrote the Deep Dive article and put everything there.

Back to basics

The problem and all the similar certificate-related issues (in any language not only python) on the internet require a clean and clear understanding of the certificate chain. I have very clearly explained everything in my deep dive post.

Typically the certificate chain consists of 3 parties.

A root certificate authority
One or more intermediate certificate authority
The server certificate is asking for the certificate to be signed.

The delegation of responsibility is:

Root CA signs → intermediate CA

Intermediate CA signs → server certificate

The Root certificates from Root CAs typically have a very long expiry date (more than 20 years) and come bundled as CA bundles in all the computers and servers and are kept very very securely under strict rules so that no one can alter them in any machine.

As Root CA are very very sacred, they need intermediary CAs to delegate responsibility to sign a server certificate when anyone asks for it by providing a CSR. These intermediaries are called Intermediate CAs. There may be multiple intermediate CAs in a certificate chain.

Solution

In our case, when we converted the cert file to PEM format we do the error,

unable to get local issuer certificate (_ssl.c:1108)

This happens for 2 reasons:

The intermediate CA certificate is not available on the server.pem file
As we are manually specifying which certificate file to use by specifying verify=server.pem*,* the python request module will not use the already existing CA bundle, rather will use the server. pem only and expects that, it contains all the certificates in the chain, the server.cer, intermediate cert, and root cert

So I manually stripped the server certificate like this:

This is done in windows but, similar things can be done in Mac or Linux.

After stripping all the certificates from the server.cer, we will have different .cer files for all the CAs. So for the above case, we will have 4 .cer files.

Root CA(Zeescalar root ca)
Intermediate CA 1(Zscalar intermediate Root CA)
Intermediate CA 2 (Zscalar intermediate Root CA)
google server .cer file

Now, all we have to do is to convert all these .cer files to .pem files and add them together to create a consolidated pem file and feed it to python requests.

So for all the cer files run the following command 4 times.

openssl x509 -in server.cer -inform DER -outform PEM  >> consolidate.pem

All we are doing it here is to create a full fledged CA bundle which has all the certificates and anyway we can do it, is just fine.

That’s it, we feed our new CA pem file to python requests and it is happy.

response = requests.post(url, files=files, headers=headers, verify='consolidate.pem')<Response [200]>

Also published here.