How to Bootstrap a Certificate Authority in your Kubernetes Cluster

In overall, I was a bit unhappy using self-signed TLS certificates in my home lab Kubernetes cluster. I found it annoying to click away these warnings in my browser. Also, I ran into a couple of problems using self-signed certificates for my Keycloak installation. When configuring SSO for ArgoCD and Grafana I had to configure security overrides when calling Keycloak OIDC endpoints for the authentication process. So I decided to create my owm certificate authority eventually.

A friend recommended the solution from Smallstep to me. These guys provide Step CA which is a piece of software which issues TLS certificates based on your own root certificate authority (CA). They also provide Step Issuer which is a Kubernetes cert-manager CertificateRequest controller that uses Step CA.

Installing Step CA and Step Issuer und configuring Ingresses with it, I fell into a couple of traps. So that process was not as straight forward as I thought and I spent a while on a working solution. So I guess it's worth sharing my experience with the public.

Step CA

I was using the Helm chart to install Step CA. As a first step you want to create your values.yaml file. Smallstep offers a CLI tool which comes in handy here. It's quickly installed on your machine. You can generate your values.yaml like so:

step ca init --helm > values.yaml

This will result in some interactive process where you need to enter the following configuration options:

1. Deployment Type: you want to select Standalone here
2. Name od the PKI: pick something that suits you
3. DNS names: here you need to have a FQDN for:
4. cert-manager (mandatory): this needs to be some FQDN that your internal Kubernetes DNS can resolve, i.e. step-certificates.security.svc.cluster.local or step-certificates.cert depending on the namespace you will install it into
5. the outside world: if you choose to offer that service on some public domain i.e. ca.yourdomain.com
6. IP and port: go with the default :9000 or pick whatever matches your use case
7. First provisioner name: as we are aiming for cert-manager using it, cert-manager is probably a good choice
8. Password: when you generate one, it ends up base64encoded in that values.yaml file

Depending on how you conduct the installation process with Helm, you might want to remove the password and certificates, keep it in your credential store and inject it later with your provisioning tool. I use OpenTofu for this. A simple install using Helm CLI can be done like this:

helm repo add smallstep https://smallstep.github.io/helm-charts/
helm install step-certificates smallstep/certificates -f values.yaml --namespace security

Please be aware of the security namespace. This will result in Step Certificates being available under the FQDN step-certificates.security.svc.cluster.local in your Kubernetes cluster. As you probably don't want to install and run it in your default namespace, this is this first trap you can fall into.

Results

Now check for ConfigMaps created by that Helm chart:

$ kubectl -n security get configmap
NAME                       DATA   AGE
kube-root-ca.crt           1      42h
step-certificates-certs    2      42h
step-certificates-config   4      42h

And have a closer look at that step-certificates-certs ConfigMap:

kubectl -n security get configmap step-certificates-certs -o yaml

Take note that includes the root_ca certificate. This will come in handy later.

Also check on the step-certificates-config ConfigMap:

kubectl -n security get configmap step-certificates-config -o yaml

Take note that it includes the configuration for the provisioner that we generated with the Step CLI tool earlier. We will need that later to configure the Step Issuer.

Check on the Secrets which were created by the Helm chart:

$ kubectl -n security get secret                                   
NAME                                      TYPE                                 DATA   AGE
sh.helm.release.v1.step-certificates.v1   helm.sh/release.v1                   1      42h
step-certificates-ca-password             smallstep.com/ca-password            1      42h
step-certificates-provisioner-password    smallstep.com/provisioner-password   1      42h
step-certificates-secrets                 smallstep.com/private-keys           2      42h

Take not that there is a secret containing the provisioner password step-certificates-provisioner-password. This we also need to configure the Step Issuer.

Step Issuer

Smallstep provides another Helm chart to install Step Issuer. I wanted to have a ClusterIssuer for my Kubernetes Cluster. Fortunately that Helm chart comes with a template to create it. But for configuring it we need the following seven values:

1. CA URL
2. CA Root certificate (base64 encoded)
3. Provisioner name
4. Provisioner Key ID (KID)
5. From the provisioner Secret (see above step-certificates-provisioner-password)
6. Name of the Secret
7. Key of the password in that Secret
8. Namespace where that Secret lives

For extracting the needed values you need to have jq installed.

CA URL can be extracted from step-certificates-config ConfigMap (see above):

kubectl -n security get configmap step-certificates-config -o jsonpath="{.data['defaults\.json']}"  | jq -r '."ca-url"'

CA root certificate can be extracted from step-certificates-certs ConfigMap (see above) and encode it with base64:

kubectl -n security get configmap step-certificates-certs -o jsonpath="{.data['root_ca\.crt']}"  | base64

Extract provisioner name from step-certificates-config ConfigMap (see above) like so:

kubectl -n security get configmap step-certificates-config -o jsonpath="{.data['ca\.json']}"  | jq -r '.authority.provisioners[0].name'

Extract provisioner KID from step-certificates-config ConfigMap (see above) like so:

kubectl -n security get configmap step-certificates-config -o jsonpath="{.data['ca\.json']}"  | jq -r '.authority.provisioners[0].key.kid'

With the values you acquired, you can put together your values.yaml file for Step Issuer Helm chart:

stepClusterIssuer:
  create: true
  caUrl: "https://step-certificates.security.svc.cluster.local"
  caBundle: "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJvekNDQVVxZ0F3SUJBZ0lSQU5zTnFuNGNLb0JYN3RLMDFPU"
  provisioner:
    name: "cert-manager"
    kid: "KCE2Wd2sJB-3adZZpPueITNIe8KyXw0Om17-kDzZ_fQ"
    passwordRef:
      name: "step-certificates-provisioner-password"
      key: "password"
      namespace: "security"

Then use that values.yaml file to install Step Issuer in your Kubernetes cluster:

helm install step-issuer smallstep/step-issuer -f values.yaml --namespace security

Results

Check the results after installation:

$kubectl get stepclusterissuer        
NAME          AGE
step-issuer   3d3h

So you should now have a StepClusterIssuer up and running. Please note that its name is step-issuer and not step-cluster-issuer. You need that name to reference it later in the Ingress annotations. For some reason I don't understand the Helm chart does not provide any way to change that. Also check its status:

kubectl get stepclusterissuer step-issuer -o yaml

If there is any problem with it, you would see it in the output.

cert-manager

You have your StepClusterIssuer up and running now. So cert-manger can use it for issuing TLS certificates. Dealing with cert-manager Ingress annotations for external issuers is a bit tricky though. Please take note of all the warnings in official cert-manager documentation. So you want to have the annotations part for an Ingress look like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/issuer: step-issuer
    cert-manager.io/issuer-group: certmanager.step.sm
    cert-manager.io/issuer-kind: StepClusterIssuer

Please note that cert-manager.io/issuer refers to the name of the StepClusterIssuer step-issuer (see above).

Import Root CA

You want to install your new root certificate in your systems trust store eventually. So browsers and other applications can deal with TLS certificates issued based on that root certificate appropriately. So it makes sense to put your new root certificate into a file i.e. root-ca.pem that you can use and share.

Local Machines

Step CLI tool offers a very convenient way to install the root certificate into your local default trust store:

step certificate install root-ca.pem

SSO with Keycloak for ArgoCD

ArgoCD offers a nice configuration option for using custom root certificates:

  oidc.config: |
    ...
    rootCA: |
      -----BEGIN CERTIFICATE-----
      ... encoded certificate data here ...
      -----END CERTIFICATE-----

SSO with Keycloak and Grafana

Grafana is also offering a configuration option for custom root certificates. Please see tls_client_ca in that section.