DNS and Certificates made simple for Kubernetes

DNS and Certificates made simple for Kubernetes

One of the challenges with demonstrating what you have done in Kubernetes is directing others to your application in a secure manner. This typically involves you having to create a DNS entry and requesting a certificate. Two projects, external-dns and cert-manager, respectively, are making it much easier to automate this process and hide the complexities from developers. Both projects are under active development (with commits in the last few hours or days) and a lot of interest (thousands of stars on GitHub).

In the post that follows we’ll create a Kubernetes cluster, add an ingress controller, and deploy our application. Then we’ll deploy external-dns and cert-manager in our cluster. Finally, we’ll show how they can be used to give end users secure access to our application.

Create a Kubernetes Cluster

To create our Kubernetes cluster, we will use VMware’s Tanzu Kubernetes Grid or TKG. TKG uses Cluster-API to manage the lifecycle of Kubernetes clusters. In our case, we have a management cluster already running in AWS. We’ll use it to create our workload cluster. The default dev plan creates a cluster with one control plan node and one worker node (as well as a bastion host for accessing them, if necessary). We’ll modify this plan to have it create two worker nodes instead of one. We then retrieve the credentials for administering the cluster and see what has been built.

$ tkg create cluster demo-cluster --plan=dev --worker-machine-count 2
$ tkg get clusters
 NAME                 NAMESPACE  STATUS   CONTROLPLANE  WORKERS  KUBERNETES        ROLES
 demo-cluster         default    running  1/1           2/2      v1.19.3+vmware.1  <none>
$ tkg get credentials demo-cluster
Credentials of workload cluster 'demo-cluster' have been saved
You can now access the cluster by running 'kubectl config use-context demo-cluster-admin@demo-cluster'
$ kubectl config use-context demo-cluster-admin@demo-cluster
Switched to context "demo-cluster-admin@demo-cluster".
$ kubectl get nodes
NAME                                       STATUS   ROLES    AGE    VERSION
ip-10-0-1-235.us-west-2.compute.internal   Ready    <none>   119m   v1.19.3+vmware.1
ip-10-0-1-239.us-west-2.compute.internal   Ready    <none>   126m   v1.19.3+vmware.1
ip-10-0-1-96.us-west-2.compute.internal    Ready    master   128m   v1.19.3+vmware.1

Install an Ingress Controller

To give end users access to applications running in our Kubernetes cluster, we need to install an ingress controller. For this example, we are going to use the Contour ingress controller. There are three options for installing Contour, a deployment manifest, an operator, or a Helm chart. We’ll use the Helm chart in this case but you may want to explore the other options. The operator is in alpha but bears watching as operators, in general, make lifecycle management of cluster add-ins easier.

$ helm repo add bitnami https://charts.bitnami.com/bitnami
"bitnami" has been added to your repositories
$ helm install contour bitnami/contour
NAME: contour
LAST DEPLOYED: Thu Apr 22 18:33:22 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get Contours's load balancer IP/hostname:
     NOTE: It may take a few minutes for this to become available.
     You can watch the status by running:
         $ kubectl get svc contour-envoy --namespace default -w
     Once 'EXTERNAL-IP' is no longer '<pending>':
         $ kubectl describe svc contour-envoy --namespace default | grep Ingress | awk '{print $3}'
2. Configure DNS records corresponding to Kubernetes ingress resources to point to the load balancer IP/hostname found in step 1

We can ignore the notes above as we will use external-dns to automate this process for us.

Install an application

For this demonstration, we’re going to install a generic NGINX website. For this, we’ll create simple Deployment and Service manifests and apply them.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
        ports:
        - containerPort: 80
apiVersion: v1
kind: Service
metadata:
  name: nginx-svc
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: ClusterIP
$ kubectl apply -f nginx-deployment.yaml 
deployment.apps/nginx created
$ kubectl apply -f nginx-service.yaml 
service/nginx-svc created

We now have our demo application running in our new Kubernetes cluster but there is no way for an end user to access it easily and securely.

Install external-dns

The external-dns project configures DNS servers with addresses for services exposed by a Kubernetes cluster. external-dns supports a large variety of DNS servers from cloud providers like AWS, Azure, and Google to more domain centric providers like Infoblox, GoDaddy, and DNSimple. Check the GitHub repository for a complete list. In our case, we use Azure DNS to manage our lab.capstonec.net subdomain.

To use external-dns, we need to create a JSON file with the details of an Azure service principal with contributor permissions for the resource group (capstonec.net in this case) which contains our DNS Zone resource (lab.capstonec.net).

{
  "tenantId": "12345678-9012-3456-7890-123456789012",
  "subscriptionId": "abcdefgh-ijkl-mnop-qrst-uvwxyzabcdef",
  "resourceGroup": "capstonec.net",
  "aadClientId": "01234abc-de56-ff78-abc1-234567890def",
  "aadClientSecret": "0VZBkxeCSOtDHGgEMftg"
}

And, create a Kubernetes secret, azure-config-file, from this JSON file.

$ kubectl create secret generic azure-config-file --from-file=azure.json
secret/azure-config-file created

Next, we create a deployment manifest for external-dns which includes the details of the DNS domain being managed and the service principal secret we created above. The manifest also includes the ClusterRole and ClusterRoleBinding which allows external-dns to listen for changes to ingresses, services, etc.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: external-dns
rules:
- apiGroups: [""]
  resources: ["services","endpoints","pods"]
  verbs: ["get","watch","list"]
- apiGroups: ["extensions","networking.k8s.io"]
  resources: ["ingresses"] 
  verbs: ["get","watch","list"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: external-dns-viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
- kind: ServiceAccount
  name: external-dns
  namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-dns
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: external-dns
  template:
    metadata:
      labels:
        app: external-dns
    spec:
      serviceAccountName: external-dns
      containers:
      - name: external-dns
        image: k8s.gcr.io/external-dns/external-dns:v0.7.6
        args:
        - --source=service
        - --source=ingress
        - --domain-filter=lab.capstonec.net
        - --provider=azure
        volumeMounts:
        - name: azure-config-file
          mountPath: /etc/kubernetes
          readOnly: true
      volumes:
      - name: azure-config-file
        secret:
          secretName: azure-config-file

Deploy external-dns.

$ kubectl apply -f external-dns.yaml 
serviceaccount/external-dns created
clusterrole.rbac.authorization.k8s.io/external-dns created
clusterrolebinding.rbac.authorization.k8s.io/external-dns-viewer created
deployment.apps/external-dns created

At this point, external-dns is now listening for ingresses and services of type LoadBalancer and will create DNS CNAME entries that correspond to them and point them to the AWS load balancer associated with our Contour ingress controller.

Install cert-manager

The cert-manager project makes the process of requesting and renewing certificates easy for resources within a Kubernetes cluster. As with the external-dns project, cert-manager supports a wide variety of certificate authorities. We are going to use the Let’s Encrypt (or ACME) issuer. The ACME issuer supports two types of challenges HTTP01 and DNS01. We will use DNS01 with our same Azure DNS server.

We will use Helm to install cert-manager into the cert-manager namespace. The process is simple as you can see below.

$ helm repo add jetstack https://charts.jetstack.io
"jetstack" has been added to your repositories
$ helm repo update 
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "jetstack" chart repository
...Successfully got an update from the "bitnami" chart repository
$ kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.2.0/cert-manager.crds.yaml
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created
$ kubectl create namespace cert-manager
namespace/cert-manager created
$ helm install cert-manager jetstack/cert-manager --namespace cert-manager
NAME: cert-manager
LAST DEPLOYED: Thu Apr 22 18:47:37 2021
NAMESPACE: cert-manager
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
cert-manager has been deployed successfully!

In order to begin issuing certificates, you will need to set up a ClusterIssuer
or Issuer resource (for example, by creating a 'letsencrypt-staging' issuer).

More information on the different types of issuers and how to configure them
can be found in our documentation:

https://cert-manager.io/docs/configuration/

For information on how to configure cert-manager to automatically provision
Certificates for Ingress resources, take a look at the `ingress-shim`
documentation:

https://cert-manager.io/docs/usage/ingress/

To use cert-manager, we need to create an Issuer or ClusterIssuer resource. The former is namespace constrained so, for ease of use, we create a ClusterIssuer for Let’s Encrypt using Azure DNS as the DNS01 solver. The information you need for DNS01 is the same information you needed for external-dns.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: azuredns-cluster-issuer
spec:
  acme:
    email: ken.rider@capstonec.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: azuredns-clusterissuer-account-key
    solvers:
    - dns01:
        azureDNS:
          clientID: 01234abc-de56-ff78-abc1-234567890def
          clientSecretSecretRef:
            name: azuredns-config
            key: client-secret
          subscriptionID: abcdefgh-ijkl-mnop-qrst-uvwxyzabcdef
          tenantID: 12345678-9012-3456-7890-123456789012
          resourceGroupName: capstonec.net
          hostedZoneName: lab.capstonec.net
          environment: AzurePublicCloud

In addition, we need to create a Kubernetes secret for the Azure service principal’s password.

$ AZURE_CERT_MANAGER_SP_PASSWORD="0VZBkxeCSOtDHGgEMftg"
$ kubectl -n cert-manager create secret generic azuredns-config --from-literal=client-secret=$AZURE_CERT_MANAGER_SP_PASSWORD
secret/azuredns-config created

Finally, we create the ClusterIssuer resource.

$ kubectl -n cert-manager apply -f azuredns-clusterissuer.yaml
clusterissuer.cert-manager.io/azuredns-cluster-issuer created

Create an ingress for our application

For an end user to access the nginx-svc service running in our cluster, we need to tell the ingress controller what HTTP traffic to route to it and how to terminate HTTPS traffic and route it as HTTP. We do this by specifying an Ingress resource.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx
  annotations:
    cert-manager.io/cluster-issuer: azuredns-cluster-issuer
spec:
  rules:
  - host: nginx.lab.capstonec.net
    http:
      paths:
      - backend:
          serviceName: nginx-svc
          servicePort: 80
        path: /
  tls:
  - hosts:
    - nginx.lab.capstonec.net
    secretName: nginx-ingress-certificate
$ kubectl apply -f nginx-ingress.yaml 
ingress.networking.k8s.io/nginx created

The ingress specifies that any HTTP or HTTPS traffic with a host header matching nginx.lab.capstonec.net be routed to the nginx-svc service. However, we don’t have a DNS entry for nginx.lab.capstonec.net at this point nor do we have a certificate for it for HTTPS termination.

This is where external-dns and cert-manager come in. They are watching all Ingress resources. If there is a host key in the rules spec that matches the DNS domain we specified in the external-dns configuration, it attempts to create a DNS CNAME record for it with a value that matches the IP address or DNS name of the ingress controller. Similarly, if there is a hosts key in the tls spec that matches the DNS zone specified in the cert-manager ClusterIssuer resource and the cert-manager.io/cluster-issuer annotation is present, it will generate a signed certificate for it and store it in the specified Kubernetes Secret for use by the ingress controller.

View our application

Let’s open our website, https://nginx.lab.capstonec.net, in our browser. Our browser uses DNS to resolve our hostname, nginx.lab.capstonec.net. If we query DNS, we can see that Azure DNS has created a CNAME entry for our hostname which points to the AWS load balancer which fronts our Contour ingress controller. Since we specified HTTPS in the URL, our browser needs to fetch the certificate for our site, verify the name it specifies corresponds to the hostname we used, and validate it is signed by a certificate authority (CA) we trust. As you can see, our site has a certificate for our hostname and it is signed by the Let’s Encrypt CA.

nginx.lab.capstonec.net in your browser

Summary

external-dns and cert-manager are just two of the many projects making Kubernetes easier to use, more automated, and more secure. If you want or need help with your Kubernetes journey, Capstone IT is a VMware Modern Applications Partner as well as being an Azure Gold and AWS Select partner. If you are interested in finding out more and getting help with your Container, Cloud, and DevOps transformation, please Contact Us.

Ken Rider

Solutions Architect
Capstone IT
Certified Kubernetes AdministratorCertified Kubernetes Application DeveloperVMware Cloud Native Master Specialist

 

 

Featured image by Andrey Sharpilo on Unsplash

Leave a Comment

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.