Getting Tomcat logs from Kubernetes pods

I have been working with a client recently on getting Tomcat access and error logs from Kubernetes pods into Elasticsearch and visible in Kibana. As I started to look at the problem and saw Elastic Stack 7.5.0 released, it also seemed like a good idea to move them up to the latest release. And, now that Helm 3 has been released and no longer requires Tiller, using the Elastic Stack Kubernetes Helm Charts to manage their installs made a lot of sense.

Dynamically Configure reCaptcha Site Key

I’ve recently been working on a new web portal and ran into a problem of trying to figure out how to dynamically configure the reCaptcha site key.

It is a basic single page application (SPA) utilizing the Angular web framework (with component based development) written using Typescript (love it!) and decorated using Bootstrap CSS (can’t live without it!). The frontend Angular application is served up from the backend Java SpringBoot. application And, the whole thing is deploy in Docker containers to a Docker Swarm cluster.

I have a lot to be thankful for

In the days leading up to the Thanksgiving Holiday weekend, I want to talk about being thankful.  This is not your typical tech blog post.

Today my wife Sharon and I both went outside into the front garden to trim back the shrubs, prune the rose bushes, and remove all the debris from our summer plants.  It was a pleasant afternoon with a cool breeze in the air.  The sun shone on my back and warmed me as I worked.  With a pair of leather gloves and some pruning shears in hand, we made short work of the fall garden cleanup.  Sharon and I enjoy our time together and especially working together on our property.

What is Container Orchestration – Kubernetes Version?

In a previous post, What is Container Orchestration?, I explained container orchestration using some examples based on Docker Swarm. While Docker Swarm is undeniably easier to both use and explain, Kubernetes is by far the most prevalent container orchestrator today. So, I’m going to go through the same examples from that previous post but, this time, use Kubernetes. One of the great things about Docker Enterprise is it supports both Swarm and Kubernetes so I didn’t have to change my infrastructure at all.

A First Look at Helm 3

Helm has been widely publicized as the package manager for Kubernetes. We’ve seen the need over and over for Helm. Unfortunately, Helm 2 requires Tiller and Tiller opens a lot of security questions. In particular, in a multi-user, multi-organization, and/or multi-tenant cluster, securing the Tiller service account (or accounts) was difficult and problematic. As a result, we’ve never recommended our clients use Helm in production. With the recent announcement of the first release candidate for Helm 3, it’s time to take another look as this version no longer requires or uses Tiller so many (most) of our security concerns should be gone.

Who Can…?

Managing a Kubernetes cluster with one user is easy. Once you go beyond one user, you need to start using Role-Based Access Control (RBAC). I’ve delved into this topic several times in the past with posts on how to Create a Kubernetes User Sandbox in Docker Enterprise and Functional Kubernetes Namespaces in Docker Enterprise. But, once you get beyond a couple of users and/or teams and a few namespaces for them, it quickly becomes difficult to keep track of who can do what and where. And, as time goes on and more and more people have a hand in setting up your RBAC, it can get even more confusing. You can and should have your RBAC resource definitions in source control but it’s not easy to read and is hard to visualize. Enter the open source who-can kubectl plugin from the folks at Aqua Security. It gives you the ability to show who (subjects) can do what (verbs) to what (resources) and where (namespaces).

Configure Custom CIDR Ranges in Docker EE

I recently worked with a customer to customize all of the default Classless Interdomain Routing (CIDR) ranges used for IP address allocation by Docker Enterprise Edition 3.0 (Docker EE 3.0). The customer primarily wanted to document the customization process for future use. However, there is often a real need to change some of the default CIDR ranges to avoid conflicts with existing private IP addresses already in use within a customer’s network. Typically such a conflict will make it impossible for applications running in containers or pods to access external hosts in the conflicting CIDR range.

Auto Scaling Docker Nodes in AWS

ButterBall Turkey

I once heard a story from someone who worked at ConAgra. They produce and sell a variety of food products that you and I eat all the time. The most notorious is the ButterBall turkey. ConAgra owned Butterball from 1990 to 2006. Every Thanksgiving holiday, so I am told, ButterBall would have to scale up their call-center as well as their website to a couple hundred web servers to handle the demand for “how to cook my turkey?” That’s a lot of hardware!

We are only a couple months away from Thanksgiving. So, what do you call a turkey on the day after Thanksgiving? Lucky. #dadjoke

So, I decided to look into how to automatically scale up my Docker worker nodes when the average CPU threshold of 80% utilization is crossed. It turns out to be pretty straight forward and can be accomplished with a three step process.

  1. Create a Launch Template
  2. Create an Auto Scaling Group
  3. Create a cpu load greater than my threshold

Let’s be clear here. I am not talking about Kubernetes dynamically scaling out my pods. I am talking about measuring the average cpu utilization across all of my worker nodes and creating a brand new virtual machine (VM) that is added to my cluster. So when ALL my containers get a bit punchy, then it’s time to expand their world.

Starting Point

Before we get started with this 3 step approach, let’s talk about my cluster. Currently my demo cluster contains 1 UCP manager, 1 DTR, and 1 Worker. By the time we are done, I will show you how it dynamically grows by one worker node.

Auto Scaling Docker Nodes in AWS

Create a Launch Template

Logging into my AWS account using the left navigation to find Instances and select the Launch Templates.

Auto Scaling Docker Nodes in AWS

You can see in the diagram above that I already have a launch template created. Let’s see what it takes to create this. By selecting the Create Launch Template button you will get the following form to fill out. There are three critical things to fill out here. First you need to select an AMI by it’s ID and it should be a known AMI that you will use for all your Docker nodes. Second, you want to choose an appropriately sized VM that suits the needs of your worker nodes. I chose a t2.medium for demonstration purposes only. Third, be sure to include your key pair so that you can login to your VM. Next you will need to choose the appropriate security groups to apply to this VM so that you can SSH into the machine as needed and allow http/https traffic.

Auto Scaling Docker Nodes in AWS

Next you will want to scroll down to the bottom of the form and open the Advanced Details. This section allows you to execute a one-time only post-create-vm script. This is the secret sauce of how my new VM joins the cluster.

Auto Scaling Docker Nodes in AWS

This script includes the download and install of the docker engine as well as the command to join the VM to the Docker cluster. Copy the script from below.

#!/bin/bash
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl software-properties-common
export DOCKER_EE_URL=https://storebits.docker.com/ee/m/sub-abcdefg-hijk-lmno-pqrst-uvwxyz/
export DOCKER_EE_VERSION=18.09
curl -fsSL "${DOCKER_EE_URL}/ubuntu/gpg" | sudo apt-key add -
sudo apt-key fingerprint 6D085F96
sudo add-apt-repository "deb [arch=$(dpkg --print-architecture)] $DOCKER_EE_URL/ubuntu $(lsb_release -cs) stable-$DOCKER_EE_VERSION"
sudo apt-get update
sudo apt-get install -y docker-ee=5:18.09.7~3-0~ubuntu-bionic docker-ee-cli=5:18.09.7~3-0~ubuntu-bionic containerd.io
sudo docker swarm join --token SWMTKN-1-abcdefg-qrstuvwxyz 172.31.7.68:2377

Create an Auto Scaling Group

Now we can create an AWS Auto Scaling Groups that utilizes your Launch template. Start by navigating to Auto Scaling and select Auto Scaling Groups. Then select the Create Auto Scaling Group button.

Auto Scaling Docker Nodes in AWS

Now choose the Launch Template panel on the right. Then select your newly created launch template from the list at the bottom and press the Next Step button.

Auto Scaling Docker Nodes in AWS

I have entered a name for my auto scaling group as well as the network and subnet’s that I want attached to my VM.

Auto Scaling Docker Nodes in AWS

This the crucial part of deciding how to to auto scale our VM’s. I have chosen an Average CPU Utilization of 75%. You can select the Review button and finish the process.

This blog doesn’t even scratch the surface of what is available for auto scaling. Read more about Auto Scaling Groups.

Simulate a Load

Now we need to test that the Auto Scaling Group will function properly. The easiest thing is to login to the existing worker node and run the progrium/stress container.

$ sudo docker run --rm -it progrium/stress --cpu 2
stress: info: [1] dispatching hogs: 2 cpu, 0 io, 0 vm, 0 hdd
stress: dbug: [1] using backoff sleep of 6000us
stress: dbug: [1] --> hogcpu worker 2 [6] forked
stress: dbug: [1] using backoff sleep of 3000us
stress: dbug: [1] --> hogcpu worker 1 [7] forked

This will immediately spin up 2 programs to stress each cpu on this VM. As you’ll notice below stress is running at a hefty pace on both cores for a total system load of 98.8% cpu utilization.

Auto Scaling Docker Nodes in AWS

This load will certainly surpass our 75% threshold for our auto scaling group. And looking at the Activity History you can see the event triggered a new VM to be provisioned.

Auto Scaling Docker Nodes in AWS

This is very exciting to me. Now we just need to wait until it is totally up and running and verify if the Docker engine was installed correctly and see if it properly joins the cluster.

The Results

Auto Scaling Docker Nodes in AWS

Well, there you have it. I now have 2 Kubernetes worker nodes in my cluster!

The beauty of Auto Scaling Groups is that you can tune many parameters including the instance type of the VM you are creating. You can even choose to use Spot Instances and get up to 90% reduced cost for your AWS VM’s.

Summary

When I have a customer facing containerized web application that will get hammered over the holidays, then I now have a solution to automatically scale out my VM’s using an AWS feature called Auto Scaling Groups. There is so much more to talk about regarding scaling including placement of new vm’s, scaling policies, scaling back, etc.

My family told me to stop telling Thanksgiving jokes, but I told them I couldn’t quit “cold turkey”. Feel free to contact us at Capstone IT especially if you want more #dadjokes. See you next time.

Mark Miller
Solutions Architect
Docker Accredited Consultant

Interlock Service Clusters – with Code!

So a colleague of mine was helping his client configure Interlock and wanted to know more about how to configure Interlock Service Clusters.  So I referred him to my previous blog – Interlock Service Clusters.  While that article conceptually helps someone understand the capabilities of Interlock, it does not show any working code examples.

Let’s review what Docker Enterprise UCP Interlock provides. And then I will show you how to configure Interlock to support multiple ingresses each of which are tied to its own environment.

Interlock Review

The Interlock ingress provides three services.

  • Interlock (I) – an overall manager of all things ingress and a listener to Swarm events. It spawns both the extension and proxy services.
  • Interlock Extension (IE) – When Interlock notices Swarm service changes it will then notify the Interlock Extension to create a new Nginx configuration file. That file is returned to the Interlock.
  • Interlock Proxy (IP) – the core ingress listener that routes traffic based on http host header to appropriate application services. It receives its Nginx configuration from Interlock whenever there are service changes that the Interlock Proxy needs to handle.

The Interlock services containers are represented in the diagram below as I for Interlock, IE for Interlock Extension, and IP for Interlock Proxy.

interlock multi service

The shaded sections represent Docker Collections for dev, test, and prod environments; all managed within the single cluster. Integrating Interlock Service clusters into this approach provides a benefit in that of isolating problems to a single collection. This is a much more fault tolerant and ensures downstream test and prod ingress traffic is unaffected.  The second benefit is that this provides greater ingress capacity for each environment. The production Interlock Proxies are dedicated for production use only and therefore does not share its capacity with dev and test ingress traffic.

We will establish 3 Interlock Service Clusters and have it deploy one ucp-interlock-proxy replica to each node that has the label of com.docker.interlock.service.cluster.

The overall process we work thru entails the following steps.

  • enable interlock
  • pulling down Interlock’s configuration toml
  • configuring three service clusters
  • upload a new configuration with a new name
  • restart the interlock service

The code that I will show you below is going to be applied to my personal cluster in AWS. In my cluster I have 1 manager, 1 dtr, and 3 worker nodes.  Each worker node is assigned to one of 3 collections named /dev, /test, and /prod. I will setup a single dedicated interlock proxy on each of these environments to segregate ingress traffic for dev, test, and prod.

$ docker node ls
ID                            HOSTNAME                            STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION
ziskz8lewtzu7tqtmx     ip-127-13-5-3.us-west-2.compute.internal   Ready     Active                          18.09.7
5ngrzymphsp4vlwww7     ip-127-13-6-2.us-west-2.compute.internal   Ready     Active                          18.09.7
qqrs3gsq6irn9meho2 *   ip-127-13-7-8.us-west-2.compute.internal   Ready     Active         Leader           18.09.7
5bzaa5xckvzi4w84pm     ip-127-13-1-6.us-west-2.compute.internal   Ready     Active                          18.09.7
kv8mocefffu794d982     ip-127-13-1-5.us-west-2.compute.internal   Ready     Active                          18.09.7

With Code

Step 1 – Verify Worker Nodes in Collections

Let’s examine the Let’s examine a worker node to determine its collection.

$ docker node inspect ip-127-13-5-3.us-west-2.compute.internal | grep com.docker.ucp.access.label
                "com.docker.ucp.access.label": "/dev",

You can repeat this command to inspect each node and determine if they reside in the appropriate collection.

Step 2 – Enable Interlock

If you have not already done so, then navigate in UCP to adminadmin settingsLayer 7 Routing.  Then select the check box to Enable Layer 7 Routing.

Interlock Service Clusters - with Code!

Step 3 – Label Ingress Nodes

Add an additional label to each node whose purpose is dedicated to running interlock proxies.

$ docker node update --label-add com.docker.interlock.service.cluster=dev ip-127-13-5-3.us-west-2.compute.internal
ip-127-13-5-3.us-west-2.compute.internal

Repeat this command for each node you are dedicating for ingress traffic and assign appropriate environment values for test and prod.

Step 4 – Download Interlock Configuration

Now that Interlock is running, let us extract its running configuration and modify that to suit our purposes.

CURRENT_CONFIG_NAME=$(docker service inspect --format '{{ (index .Spec.TaskTemplate.ContainerSpec.Configs 0).ConfigName }}' ucp-interlock)
docker config inspect --format '{{ printf "%s" .Spec.Data }}' $CURRENT_CONFIG_NAME > config.orig.toml

The default configuration for Interlock is to have two interlock proxies running anywhere in the cluster. The proxies configuration resides in a section named Extensions.default. This is the heart of an interlock service cluster. We will duplicate this section two times for a total of three sections and then rename them to suit our needs.

 

Step 5 – Edit Interlock Configuration

Copy the config.orig.toml file to config.new.toml. Then, using your favorite editor (vi of course) duplicate the Extensions.default section two more times.  Rename each of the three Extension.defaults to Extension.dev, Extensions.test, and Extensions.prod.  Each Extensions.<env> section has other sub-sections that include the same name plus a qualifier (e.g. Extensions.default.Config).  These too will need to be renamed.

Now we have 3 named extensions for each of dev, test, and prod.  Next, you will search for the PublishedSSLPort and change it to 8445 for dev, and 8444 for test and leave the value 8443 for prod.  These 3 ports should be the values that the incoming load balancer uses in its back-end pools. For each environment specific VIP (dev, test, prod) the traffic will flow into the load balancer on port 443.  The VIP used to access the load balancer will dictate how the traffic will be routed to the appropriate interlock proxy IP address and port.  

Add a new property called ServiceCluster under each of the extensions sections and give it the name of dev, test, or prod.

You can also specify the constraint labels that will dictate where both the Interlock Extension and Interlock Proxies will run. Start by changing the Constraints and ProxyConstraints to use your new node labels.

The ProxyReplicas indicates how many container replicas to run for the interlock proxy service. We will set ours to 2. The ProxyServiceName is the name of the service as it is deployed into Swarm for this service. We will name ours ucp-interlock-proxy-dev which is specific to the environment it is supporting.

Of course you will do this for all three sections within the new configuration file. Below is a snippet of only the changes that I have made for the dev ingress configuration. You will want to repeat this for test and prod as well.

  [Extensions.dev]
    ServiceCluster = "dev"
    ProxyServiceName = "ucp-interlock-proxy-dev"
    ProxyReplicas = 1
    PublishedPort = 8082
    PublishedSSLPort = 8445
    Constraints = ["node.labels.com.docker.interlock.service.cluster=dev"]
    ProxyConstraints = ["node.labels.com.docker.interlock.service.cluster=dev"]

This snipped of the configuration file only represents the values that have changed. There are numerous others you should just leave as is.

Step 6 – Upload new Interlock Configuration

NEW_CONFIG_NAME="com.docker.ucp.interlock.conf-$(( $(cut -d '-' -f 2 <<< "$CURRENT_CONFIG_NAME") + 1 ))"
docker config create $NEW_CONFIG_NAME config.new.toml

Step 7- Restart Interlock Service with New Configuration

docker service update --update-failure-action rollback \
    --config-rm $CURRENT_CONFIG_NAME \
    --config-add source=$NEW_CONFIG_NAME,target=/config.toml \
    ucp-interlock
ucp-interlock
overall progress: 1 out of 1 tasks 
1/1: running   [==================================================>] 
verify: Service converged 
interlock-service-cluster-config

Note: in the above scenario the service update worked smoothly. Other times, such as when there are errors in your configuration, the service will rollback. In those cases you will want to do a docker ps -a | grep interlock and look for the recently exited docker/ucp-interlock container. Once you have its container id you can perform a docker logs <container-id> to see what went wrong.

Step 8 – Verify Everything is Working

We need to make sure that everything started up properly and are listening on their appropriate ports.

docker service ls
ID                  NAME                           MODE                REPLICAS            IMAGE                                  PORTS
y3jg0mka0w7b        ucp-agent                      global              4/5                 docker/ucp-agent:3.1.9                 
xdf9q5y4dev4        ucp-agent-win                  global              0/0                 docker/ucp-agent-win:3.1.9             
k0vb1yloiaqu        ucp-auth-api                   global              0/1                 docker/ucp-auth:3.1.9                  
ki8qeixu12d4        ucp-auth-worker                global              0/1                 docker/ucp-auth:3.1.9                  
nyr40a0zitbt        ucp-interlock                  replicated          0/1                 docker/ucp-interlock:3.1.9             
ewwzlj198zc2        ucp-interlock-extension        replicated          1/1                 docker/ucp-interlock-extension:3.1.9   
yg07hhjap775        ucp-interlock-extension-dev    replicated          1/1                 docker/ucp-interlock-extension:3.1.9   
ifqzrt3kw95p        ucp-interlock-extension-prod   replicated          1/1                 docker/ucp-interlock-extension:3.1.9   
l6zg39sva9bb        ucp-interlock-extension-test   replicated          1/1                 docker/ucp-interlock-extension:3.1.9   
xkhrafdy3czt        ucp-interlock-proxy-dev        replicated          1/1                 docker/ucp-interlock-proxy:3.1.9       *:8082->80/tcp, *:8445->443/tcp
wpelftw9q9co        ucp-interlock-proxy-prod       replicated          1/1                 docker/ucp-interlock-proxy:3.1.9       *:8080->80/tcp, *:8443->443/tcp
g23ahtsxiktx        ucp-interlock-proxy-test       replicated          1/1                 docker/ucp-interlock-proxy:3.1.9       *:8081->80/tcp, *:8444->443/tcp

You can see there are 3 new ucp-interlock-extension-<env> containers and 3 new ucp-interlock-proxy-<env> containers. You can also verify that they are listening on SSL port 8443 thru 8445.  This is fine for a demonstration, but you will more than likely want to set the replica’s somewhere in the 2 to 5 range per environment.  And of course you will determine that based on your traffic load.

NOTE: Often times after the update of the Interlock’s configuration you will still see the old ucp-interlock-extension and/or the ucp-interlock-proxy services still running. You can run the following command to remove these as they are no longer necessary.

docker service rm ucp-interlock-extension ucp-interlock-proxy

Step 9 – Deploy an Application

Now let’s deploy a demo service that we can route thru our new ingress. We’re going to take the standard docker demo application and deploy it to our dev cluster. Start by creating the following docker-compose.yml file:

version: "3.7"
  
services:
  demo:
    image: ehazlett/docker-demo
    deploy:
      replicas: 2
      labels:
        com.docker.lb.hosts: ingress-demo.lab.capstonec.net
        com.docker.lb.network: ingress_dev
        com.docker.lb.port: 8080
        com.docker.lb.service_cluster: dev
        com.docker.ucp.access.label: /dev
    networks:
      - ingress_dev

networks:
  ingress_dev:
    external: true

Note that the com.docker.lb.network attribute is set to ingress_dev. I previously created this network outside of the stack. We will now utilize this network for all our ingress traffic from Interlock to our docker-demo container.

$ docker network create ingress_dev --label com.docker.ucp.access.label=/dev --driver overlay
mbigo6hh978bfmpr5o9stiwdc

Also notice that the com.docker.lb.hosts attribute is set to ingress-demo.lab.capstonec.net. I logged into our DNS server and create a CNAME record with that name pointing to my AWS load balancer for the dev environment.

I also must configure my AWS load balancer to allow traffic to a Target Group of virtual machines. We can talk about your cloud configuration in another article down the road.

Let’s deploy that stack:

docker stack deploy -c docker-stack.yml demodev

Once the stack is deployed, we can verify that the services are running on the correct machine:

docker stack ps demodev
ID                  NAME                     IMAGE                         NODE                            DESIRED STATE       CURRENT STATE           ERROR               PORTS
i3bght0p5d0j        demodev_demo.1       ehazlett/docker-demo:latest   ip-127-13-5-3.ec2.internal   Running             Running 10 hours ago                        
cyqfu0ormnn8        demodev_demo.2       ehazlett/docker-demo:latest   ip-127-13-5-3.ec2.internal   Running             Running 10 hours ago                        

Finally we should be able to open a browser to http://ingress-demo.lab.capstonec.net which routes thru the dev interlock service cluster) and see the application running.

Interlock Service Clusters - with Code!

You can also run a curl command to verify:

curl ingress-demo.lab.capstonec.net
<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="utf-8">
        <title></title>
...

Summary

Well that was a decent amount of work but now you’re done. You’ve successfully implemented your first interlock service cluster which is highly available and segmented into three environments for dev, test, and prod!

As always if you have any questions or need any help please contact us.

Mark Miller
Solutions Architect
Docker Accredited Consultant

Attack of the Kubernetes Clones

One of the customers I support is using Kubernetes under Docker EE UCP (Enterprise Edition Universal Control Plane) and has been very impressed with its stability and ease of management. Recently, however, a worker node that had been very stable for months started evicting Kubernetes pods extremely frequently, reporting inadequate CPU resources. Our DevOps team was still experimenting with determining resource requirements for many of their containerized apps, so at first, we thought the problem was caused by resource contention between pods running on the node.