Month: December 2019

Kubernetes Series – Pods (part 1)

A couple weeks back I took the test for Certified Kubernetes Application Developer developed by the Cloud Native Computing Foundation (CNCF), in collaboration with The Linux Foundation. For me personally it was satisfying to complete the test and become certified. 

Today’s blog will be the first in a series to share with you all that I have learned about Kubernetes and help you on your journey to understanding this container orchestration tool.

A Pod Story

A few years back I worked onsite for a customer in the transportation industry.  They developed a unique work environment for their IT staff. Some of their leadership team had visited other IT shops in Omaha to discover a best of breed in workspace environments.  They wanted to find the right balance between having individual privacy and yet conducive to collaboration among team members.  What they decided to do became intimately known as a pod.

Hand drawn representation of a pod which hosted 4 developers on a team.

A pod was the expansion of a cubicle to suit 4 people. There was one entrance/exit to the pod.  Each person sat facing the corner of the pod with a typical desktop and overhead shelving units.  These cubicle-pods were used as a unit-of-work in that teams of developers were situated together in a single pod.  If a team was larger, then they might occupy two pods.  But the point is that they were co-located and individuals could easily swivel in their chair and talk to another pod member. Each pod had a certain amount of privacy in that the walls were 5 feet tall, had 4 sides with an entryway on one side that opened to the walkway.

My pod-mates and I became a tight-knit team. We worked in proximity to deliver on a project assigned to us. And, due to proximity learned a bit about each other personally. So I’ve told a story about four peas in a pod; rather four developers in a large cubicle. In Kubernetes we have an analogous situation with multiple containers in a pod.

A Kubernetes Pod

Kubernetes came on the scene about 5 years ago and also uses the term pod.  A Kubernetes Pod is the smallest unit of work that can be deployed to a Kubernetes host.  Each Pod may be composed of one or more containers that collaborate together to accomplish a particular task. Pods operate a step above individual containers. It is the smallest unit that you can deploy, monitor, and observe its runtime in a Kubernetes environment.

Visualizing a Multi-Container Pod

Many Pods consist of a single container, so the newbie can generally substitute the word ‘pod’ with ‘container’ in most cases.  However, it is not uncommon to group containers in a Pod so that co-located together they process a set of work.

So a Pod has many characteristics and capabilities but the most prominent is the communication and data sharing amongst its containers.

  • Each Pod has its own IP Address and hostname (which is the Pod name). By default, all other Pods are accessible on this network.
  • A Pod’s containers share the same network and ports and communicate directly with each other on localhost.
  • A Pod uses storage volumes to enable data survival upon container restarts while also providing data sharing among the containers within the Pod.
A Multi-Container Pod with Storage and Networking

All Pods have a life-cycle that is managed by the Kubernetes orchestrator.

  • A Pod lives within a namespace. The `default` namespace is used when none is specified.
  • Fault tolerance is provided via multiple Pod replicas. Any single Pod outage is overcome by its peer replicas.
  • High availability is provided thru horizontal scaling of Pods during peak usage.
  • Pods are stopped and started by the Kubernetes orchestrator to manage a zero-downtime application rollout.
  • The orchestrator handles Pod’s colocation (co-scheduling), shared fate (e.g. termination), coordinated replication, resource sharing, and dependency management.
Pod Life Cycle Managed by Kubernetes Orchestrator

Pod Do’s and Don’ts

Pods are not meant to be full application stacks (e.g. LAMP). If your pod is an entire application, then you cannot scale up just one container, but instead are forced to scale up the entire application which defeats one of the advantages of containers.

Individual Pods are not intended to run multiple instances of the same application.  Rather, their primary motivation is to support co-located, co-managed containerized applications.

A pod can hold multiple co-located containers which are tightly coupled to share system resources. A pod typically contains one primary container and zero or more auxiliary containers. The auxiliary containers support the primary container and work as a single but cohesive unit of work. The pod then is scalable.

One example includes a log forwarding container which supports a web application container.

Example of side-car pod

Another example is a web server container that serves its content to external users with an accompanying container that manages the content provided to the webserver.

Example of a Multi-Container Pod

The previous diagram shows a multi-container Pod that contains a file puller and a web server that uses a persistent volume for shared storage between the containers.

For a detailed explanation of various Pod design patterns see Patterns for Composite Containers.

The Manifest (YAML)

The pod is describe using a YAML manifest file.  The apiVersion, kind, metadata, and spec are required fields.

  • apiVersion – version of the Kubernetes API
  • kind – the kind of object to create, such as Pod
  • metadata – any information to identify the pod, such as a name or namespace
  • spec – details of our pods containers and other configuration

A Single-Container Pod

The following manifest shows a single container pod with a a simple nginx web server listening on port 80.

apiVersion: v1
kind: Pod
  name: web-pod
    app: myapp
  - name: web-container
    image: nginx
    - containerPort: 80

You can use kubectl to create the pod using our yaml manifest and to obtain its running status.

kubectl create -f web-pod.yml

kubectl get pod

A Multi-Container Pod

A multi-container pod manifest can be seen below.  The first primary difference from the single container pod is a second element under the containers array. The second difference is that both containers now mount a volume named shared-numbers.  That volume is declared below the containers under volumes. It is simply a temporary directory shared by both containers.

apiVersion: v1
kind: Pod
  name: webpod
  - name: web-container
    image: nginx
    - containerPort: 80
    - name: shared-numbers
      mountPath: /usr/share/nginx/html
  - name: number-generator
    image: alpine 
    command: ["/bin/sh"]
    args: ["-c", "while true; do shuf -i 1-40 -n 3 >> /var/log/numbers.txt; sleep 10;done"]
    - name: shared-numbers
      mountPath: /var/log
  - name: shared-numbers
    emptyDir: {}


We are moving pretty fast now and have already covered a lot of ground. We know what a Pod is and what it does. We’ve seen examples of types of multi-container pods designs. And we have defined a pod using with a YAML manifest and deployed it from the command line with kubectl.

The next blog post will continue with Pod’s and more examples of how to create manifests and deploy them using the kubectl command line.

As always feel free to contact us at Capstone IT.

Mark Miller
Docker Accredited Consultant
Certified Kubernetes Application Developer


Kubeman on aisle K8S

I rarely go to Walmart and I definitely never thought I’d be going to Walmart for technology tools, advice, etc. However, thanks to Aymen EL Amri’s great curated weekly email, Kaptain, on all things Kubernetes, see, I ran across a great tool from Walmart Labs called Kubeman. If you are managing multiple Kubernetes clusters, and that’s almost always the case if you’re using Kubernetes, Kubeman is a tool you need to consider for troubleshooting. In addition to making it easier to investigate issues across multiple Kubernetes clusters, it understands an Istio service mesh as well.

Getting Tomcat logs from Kubernetes pods

I have been working with a client recently on getting Tomcat access and error logs from Kubernetes pods into Elasticsearch and visible in Kibana. As I started to look at the problem and saw Elastic Stack 7.5.0 released, it also seemed like a good idea to move them up to the latest release. And, now that Helm 3 has been released and no longer requires Tiller, using the Elastic Stack Kubernetes Helm Charts to manage their installs made a lot of sense.

Dynamically Configure reCaptcha Site Key

I’ve recently been working on a new web portal and ran into a problem of trying to figure out how to dynamically configure the reCaptcha site key.

It is a basic single page application (SPA) utilizing the Angular web framework (with component based development) written using Typescript (love it!) and decorated using Bootstrap CSS (can’t live without it!). The frontend Angular application is served up from the backend Java SpringBoot. application And, the whole thing is deploy in Docker containers to a Docker Swarm cluster.

Mind you, I have been a developer for longer than my esteemed colleague, who is working on the project with me, has even been alive. I have been a Java developer for over two decades. So, finally we have a component oriented and type safe web UI framework for developing HTML SPA applications that I actually like. Well, technically this stuff has been around for a few years; but 5 years ago working with Angular 1.4 was painful. My IntelliJ IDE has phenomenal IntelliSense for both Angular with Typescript and Java with Spring. So basically what I’m saying is that there is a new full-stack developer in town.

Google reCaptcha

There is one particular page we were concerned about ensuring that bot traffic is denied access as well as our REST API. This required us to integrate Google’s reCaptcha component and setup a reCaptcha site key and secret key thru their admin console.

recaptcha statistics from our development site
reCaptcha stats from our development site

After configuring the reCaptcha token we were able to test its capabilities and over time we gained insights thru its statistics.

Our Design

We generate a reCaptcha token on the web page and pass it along to the backend thru a REST call. The backend Java code calls Google again with the token, the site key, and the secret key and get’s back a number. This number is in the range of 0 to 1. Typically we see a 0.9 returned which is above the 0.5 threshold for determining whether it was a user or a bot that created this traffic.

web sequence diagram with Google reCaptcha
web sequence diagram with Google reCaptcha

The reCaptcha from Google uses both a site key and a secret key. The reCaptcha secret key is stored only on the backend. Whereas the site key is used on both the Angular web-app and the Java application. The Angular frontend calls Google with the reCaptcha site key to get the reCaptcha token. The backend Java code makes calls to Google with both the reCaptcha site key and secret key to get the reCaptcha number.

The Problem

We use Docker Secrets to store sensitive data like the reCaptcha secret key. We use Docker Configurations to store other configurable properties like the reCaptcha site key. And, we also had the reCaptcha site key embedded in an Angular provider in the app.modules.ts file. This was statically embedded. So the problem here is that our devops folks only want to maintain the reCaptcha site key in one place.

In the my mind this was a simple problem. We can definitely have a public facing REST service provide the reCaptcha site key. It’s already in the Angular code which is available to the world, so serving it in a REST call is no more insecure. However, the challenge began with the Angular configuration.

Remember how I told you the reCaptcha was configured using an Angular provider? The following code is a snippet of online documentation on how to configure reCatcha statically.

Using Angular provider to establish the reCaptcha Site Key
Using Angular provider to establish the reCaptcha Site Key

That’s all fine and dandy with a hard wired site key. But my devops folks do not like it. They want to maintain the reCaptcha site key on the backend, not in the frontend web app.

The Solution

Now I have to introduce a value provided from a dynamic REST call. As you can see below the fetch() method invokes the backend REST endpoint to obtain the reCaptcha site key.

Code snippet that dynamically configures the reCaptcha site key in the main.ts file
Code snippet that dynamically configures the reCaptcha site key in the main.ts file

Then the reCaptcha site key is used in the call to platformBrowserDynamic(). You can see how it is passed in as a provider in the form of a JSON array of configuration maps.

[{provide: RECAPTCHA_V3_SITE_KEY, useValue: siteKey}]

That bit of code is how I dynamically configure the reCaptcha site key, so I thought it might provide some insight for the next person. You can always get some additional help thru Capstone IT. Come check us out.

Mark Miller
Solutions Architect
Docker Accredited Consultant
(and Java/Angular developer!)