Add Self Signed or any TLS Certificate in Kubernetes POD or container’s trusted CA root certificate store

Recently I have been working on a challenge related to one cloud component which has a self signed certificate and as that certificate as well as any other certificate in its certificate chain is not available as trusted CA root Certificate , thus my application running in POD was ending up with invalid certificate issue.

To be more specific my app was consuming redis cache for a well known Public cloud and that was using Lets encrypt certificate which had a certificate chain having DST Root CA X3 certificate and thus i never face any problem as that was added as trusted root CA cert in my machine as well as default Debian GNU/Linux 9 base image for dotnet core docker image.

Now , That cloud component got upgrade and its new version started using Self signed certificate from my cloud provider with no chain certificate as any of my existing root CA certificate , thus my applications which has been working so far , started getting run-time errors during connection creation via component client.

Here example component clients can be like :

1. StackExchange.Redis nuget’s ConnectionMultiplexer object while using new Redis component (which has self signed cert)
2. RabbitMQ dotnet library object while consuming new RabbitMQ(which has self signed cert)
3. System.Net Libraries HttpClient object while calling Some API endpoint (which has self signed cert)

If you ran into something simillar problem , then this post is the right destination for solution.

To make things clear , we are assuming that you either have your certificate in any one of this format as physical file :

1. base 64 encoded or decoded .pem or .crt data (file or string) , in decoded form this looks like :

-----BEGIN CERTIFICATE-----
MIIDDXXXXXXBAgIJANEH58y2/
...........
..........XXXXaUA==
-----END CERTIFICATE-----

1. X509Certificate / X509Certificate2 format full certificate (having thumbprint , issuer etc details ) -exported as physical file.

So , to resolve this certificate validation / Invalid certificate issue , here are some possible solutions and we can use any one of them based on our convenience :

1. via Only few lines of Code Change in Application.
2. via Only few additional lines in Dockerfile while building container image.
3. via Only Kubernetes Deployment yaml/Helm chart changes while deploying Pod.

Lets us go through implementation all above options one by one with Pros and Cons.

Method 1 -Code Change :

Most of the time this kind of component client object has another callback or event used for certificate validation , for example

Problem with this approach is , it is not programming language neutral and you will need to know about specific implementation in specific programming language.

Second thing, you will end up making config changes plus code changes and reciprocate for dependent components.

Here are few gist for these examples.

HTTPClient

Redis Client

RabbitMQ Client :

Method 2 — Dokerfile Change :

This method has a benefit that it is not language dependent and also directly adds the certificate to trusted root CA certs of container and developer doesn't need to put any effort , also if your Docker image creation is also part of CI/CD , then you can plug it there.

add following lines at appropriate place in Dokerfile by copying the .pem/.crt cert text data into linux CA root store.

ADD your_ca_root.pem /usr/local/share/ca-certificates/foo.pem
RUN chmod 644 /usr/local/share/ca-certificates/foo.pem 
RUN update-ca-certificates

from above certificate will be created in container’s root CA cert which for most of the OS (Alpine / Debian / Ubuntu / Gentoo) happens to be /etc/ssl/certs/ directory. Once update-ca-certificates command runs , you will see that .pem file is added in this directory and few other symbolic links (.0 extension files) for your certificate.

Interesting thing is OS also maintains one common CA pem file at /etc/ssl/certs/ca-certificates.crt thus you will see that your .pem data got appended into this file. That’s it , you do not need to do anything else . Any application running in this container will automatically trust this certificate.

Method 3— Kubernets Deployment Yaml / Helm changes:

Sometime , it may happen that you can not plug in .pem data in the Dockerfile and can not run that update-ca-certificates as part of CI/CD.

In that case you can go at higher level and directly make changes to your Kubernetes deployment file or Helm chart with following steps

(we are showing an example with config map, but same concept can be used with volume mount which is more useful as there you can maintain all such certificates in single volume) :

1.Create config map using .pem file

kubectl -n <namespace-for-config-map-optional> create configmap ca-pemstore — from-file=my-cert.pem

2. Now , mount that config map’s file as one to one file relationship in volume mount in directory /etc/ssl/certs/ as file for example :

apiVersion: v1 
kind: Pod
metadata:
  name: cacheconnectsample
spec:
      containers:
      - name: cacheconnectsample
        image: cacheconnectsample:v1
        volumeMounts:
        - name: ca-pemstore
          mountPath: /etc/ssl/certs/my-cert.pem
          subPath: my-cert.pem
          readOnly: false
        ports:
        - containerPort: 80
        command: [ "dotnet" ]
        args: [ "cacheconnectsample.dll" ]
      volumes:
      - name: ca-pemstore
        configMap:
          name: ca-pemstore

what this will do is along with all exiting certificates in this CA root directory of pod , it will add your .pem file as well , it is partially similar to update-ca-certificates command , except that no symbolic links were created and no certificate text was appended in ca-certificates.crt , but thats file , it will still work same way an no additional changes are required.

Note : If you do not map file to file via config map but map volume to directory in yaml, then you will end up mounting config map as directory to /etc/ssl/certs/ which will add your .pem file but will wipe out all existing certificate from store .

Bingo!! you did it without modifying code or creating new application version and just putting additional responsibility on DevOps ;)

You can read in detail about same in my next story at :

Add SSL / TLS Certificate or .PEM file to Kubernetes’ Pod’s trusted root CA store

One Tricky alternate way (just for knowledge purpose , not good as compared to above approach):

1. you can copy exiting ca-certificates.crt from vanilla base linux image being used by your container.
2. append your certificate / .pem data into it and save ca-certificates.crt
3. create config map form file ca-certificates.crt
4. map that file directly to /etc/ssl/certs/ca-certificates.crt , that also works

Note : path for CA root Certificate store may vary based on OS distribution.

I hope this post was helpful , there are N number of ways we can further improve this thing by managing how config maps will be updated , multiple certificates scenario , call back events etc, the more effort you put , more optimized solutions are.

References :

https://stackoverflow.com/questions/42292444/how-do-i-add-a-ca-root-certificate-inside-a-docker-image
https://support.cloudbees.com/hc/en-us/articles/360018267271-Deploy-Self-Signed-Certificates-in-Masters-and-Agents
https://packages.debian.org/sid/ca-certificates
https://github.com/paraspatidar/Code-Dump/blob/master/Accept%20Server%20Certificate%20while%20making%20HTTP%20calls/codeSnippet.cs