What is Container Orchestration – Kubernetes Version?

In a previous post, What is Container Orchestration?, I explained container orchestration using some examples based on Docker Swarm. While Docker Swarm is undeniably easier to both use and explain, Kubernetes is by far the most prevalent container orchestrator today. So, I’m going to go through the same examples from that previous post but, this time, use Kubernetes. One of the great things about Docker Enterprise is it supports both Swarm and Kubernetes so I didn’t have to change my infrastructure at all.

Configure Custom CIDR Ranges in Docker EE

I recently worked with a customer to customize all of the default Classless Interdomain Routing (CIDR) ranges used for IP address allocation by Docker Enterprise Edition 3.0 (Docker EE 3.0). The customer primarily wanted to document the customization process for future use. However, there is often a real need to change some of the default CIDR ranges to avoid conflicts with existing private IP addresses already in use within a customer’s network. Typically such a conflict will make it impossible for applications running in containers or pods to access external hosts in the conflicting CIDR range.

Auto Scaling Docker Nodes in AWS

ButterBall Turkey

I once heard a story from someone who worked at ConAgra. They produce and sell a variety of food products that you and I eat all the time. The most notorious is the ButterBall turkey. ConAgra owned Butterball from 1990 to 2006. Every Thanksgiving holiday, so I am told, ButterBall would have to scale up their call-center as well as their website to a couple hundred web servers to handle the demand for “how to cook my turkey?” That’s a lot of hardware!

We are only a couple months away from Thanksgiving. So, what do you call a turkey on the day after Thanksgiving? Lucky. #dadjoke

So, I decided to look into how to automatically scale up my Docker worker nodes when the average CPU threshold of 80% utilization is crossed. It turns out to be pretty straight forward and can be accomplished with a three step process.

  1. Create a Launch Template
  2. Create an Auto Scaling Group
  3. Create a cpu load greater than my threshold

Let’s be clear here. I am not talking about Kubernetes dynamically scaling out my pods. I am talking about measuring the average cpu utilization across all of my worker nodes and creating a brand new virtual machine (VM) that is added to my cluster. So when ALL my containers get a bit punchy, then it’s time to expand their world.

Starting Point

Before we get started with this 3 step approach, let’s talk about my cluster. Currently my demo cluster contains 1 UCP manager, 1 DTR, and 1 Worker. By the time we are done, I will show you how it dynamically grows by one worker node.

Create a Launch Template

Logging into my AWS account using the left navigation to find Instances and select the Launch Templates.

You can see in the diagram above that I already have a launch template created. Let’s see what it takes to create this. By selecting the Create Launch Template button you will get the following form to fill out. There are three critical things to fill out here. First you need to select an AMI by it’s ID and it should be a known AMI that you will use for all your Docker nodes. Second, you want to choose an appropriately sized VM that suits the needs of your worker nodes. I chose a t2.medium for demonstration purposes only. Third, be sure to include your key pair so that you can login to your VM. Next you will need to choose the appropriate security groups to apply to this VM so that you can SSH into the machine as needed and allow http/https traffic.

Next you will want to scroll down to the bottom of the form and open the Advanced Details. This section allows you to execute a one-time only post-create-vm script. This is the secret sauce of how my new VM joins the cluster.

This script includes the download and install of the docker engine as well as the command to join the VM to the Docker cluster. Copy the script from below.

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl software-properties-common
export DOCKER_EE_URL=https://storebits.docker.com/ee/m/sub-abcdefg-hijk-lmno-pqrst-uvwxyz/
export DOCKER_EE_VERSION=18.09
curl -fsSL "${DOCKER_EE_URL}/ubuntu/gpg" | sudo apt-key add -
sudo apt-key fingerprint 6D085F96
sudo add-apt-repository "deb [arch=$(dpkg --print-architecture)] $DOCKER_EE_URL/ubuntu $(lsb_release -cs) stable-$DOCKER_EE_VERSION"
sudo apt-get update
sudo apt-get install -y docker-ee=5:18.09.7~3-0~ubuntu-bionic docker-ee-cli=5:18.09.7~3-0~ubuntu-bionic containerd.io
sudo docker swarm join --token SWMTKN-1-abcdefg-qrstuvwxyz

Create an Auto Scaling Group

Now we can create an AWS Auto Scaling Groups that utilizes your Launch template. Start by navigating to Auto Scaling and select Auto Scaling Groups. Then select the Create Auto Scaling Group button.

Now choose the Launch Template panel on the right. Then select your newly created launch template from the list at the bottom and press the Next Step button.

I have entered a name for my auto scaling group as well as the network and subnet’s that I want attached to my VM.

This the crucial part of deciding how to to auto scale our VM’s. I have chosen an Average CPU Utilization of 75%. You can select the Review button and finish the process.

This blog doesn’t even scratch the surface of what is available for auto scaling. Read more about Auto Scaling Groups.

Simulate a Load

Now we need to test that the Auto Scaling Group will function properly. The easiest thing is to login to the existing worker node and run the progrium/stress container.

$ sudo docker run --rm -it progrium/stress --cpu 2
stress: info: [1] dispatching hogs: 2 cpu, 0 io, 0 vm, 0 hdd
stress: dbug: [1] using backoff sleep of 6000us
stress: dbug: [1] --> hogcpu worker 2 [6] forked
stress: dbug: [1] using backoff sleep of 3000us
stress: dbug: [1] --> hogcpu worker 1 [7] forked

This will immediately spin up 2 programs to stress each cpu on this VM. As you’ll notice below stress is running at a hefty pace on both cores for a total system load of 98.8% cpu utilization.

This load will certainly surpass our 75% threshold for our auto scaling group. And looking at the Activity History you can see the event triggered a new VM to be provisioned.

This is very exciting to me. Now we just need to wait until it is totally up and running and verify if the Docker engine was installed correctly and see if it properly joins the cluster.

The Results

Well, there you have it. I now have 2 Kubernetes worker nodes in my cluster!

The beauty of Auto Scaling Groups is that you can tune many parameters including the instance type of the VM you are creating. You can even choose to use Spot Instances and get up to 90% reduced cost for your AWS VM’s.


When I have a customer facing containerized web application that will get hammered over the holidays, then I now have a solution to automatically scale out my VM’s using an AWS feature called Auto Scaling Groups. There is so much more to talk about regarding scaling including placement of new vm’s, scaling policies, scaling back, etc.

My family told me to stop telling Thanksgiving jokes, but I told them I couldn’t quit “cold turkey”. Feel free to contact us at Capstone IT especially if you want more #dadjokes. See you next time.

Mark Miller
Solutions Architect
Docker Accredited Consultant

Interlock Service Clusters – with Code!

So a colleague of mine was helping his client configure Interlock and wanted to know more about how to configure Interlock Service Clusters.  So I referred him to my previous blog – Interlock Service Clusters.  While that article conceptually helps someone understand the capabilities of Interlock, it does not show any working code examples.

Let’s review what Docker Enterprise UCP Interlock provides. And then I will show you how to configure Interlock to support multiple ingresses each of which are tied to its own environment.

Interlock Review

The Interlock ingress provides three services.

  • Interlock (I) – an overall manager of all things ingress and a listener to Swarm events. It spawns both the extension and proxy services.
  • Interlock Extension (IE) – When Interlock notices Swarm service changes it will then notify the Interlock Extension to create a new Nginx configuration file. That file is returned to the Interlock.
  • Interlock Proxy (IP) – the core ingress listener that routes traffic based on http host header to appropriate application services. It receives its Nginx configuration from Interlock whenever there are service changes that the Interlock Proxy needs to handle.

The Interlock services containers are represented in the diagram below as I for Interlock, IE for Interlock Extension, and IP for Interlock Proxy.

interlock multi service

The shaded sections represent Docker Collections for dev, test, and prod environments; all managed within the single cluster. Integrating Interlock Service clusters into this approach provides a benefit in that of isolating problems to a single collection. This is a much more fault tolerant and ensures downstream test and prod ingress traffic is unaffected.  The second benefit is that this provides greater ingress capacity for each environment. The production Interlock Proxies are dedicated for production use only and therefore does not share its capacity with dev and test ingress traffic.

We will establish 3 Interlock Service Clusters and have it deploy one ucp-interlock-proxy replica to each node that has the label of com.docker.interlock.service.cluster.

The overall process we work thru entails the following steps.

  • enable interlock
  • pulling down Interlock’s configuration toml
  • configuring three service clusters
  • upload a new configuration with a new name
  • restart the interlock service

The code that I will show you below is going to be applied to my personal cluster in AWS. In my cluster I have 1 manager, 1 dtr, and 3 worker nodes.  Each worker node is assigned to one of 3 collections named /dev, /test, and /prod. I will setup a single dedicated interlock proxy on each of these environments to segregate ingress traffic for dev, test, and prod.

$ docker node ls
ID                            HOSTNAME                            STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION
ziskz8lewtzu7tqtmx     ip-127-13-5-3.us-west-2.compute.internal   Ready     Active                          18.09.7
5ngrzymphsp4vlwww7     ip-127-13-6-2.us-west-2.compute.internal   Ready     Active                          18.09.7
qqrs3gsq6irn9meho2 *   ip-127-13-7-8.us-west-2.compute.internal   Ready     Active         Leader           18.09.7
5bzaa5xckvzi4w84pm     ip-127-13-1-6.us-west-2.compute.internal   Ready     Active                          18.09.7
kv8mocefffu794d982     ip-127-13-1-5.us-west-2.compute.internal   Ready     Active                          18.09.7

With Code

Step 1 – Verify Worker Nodes in Collections

Let’s examine the Let’s examine a worker node to determine its collection.

$ docker node inspect ip-127-13-5-3.us-west-2.compute.internal | grep com.docker.ucp.access.label
                "com.docker.ucp.access.label": "/dev",

You can repeat this command to inspect each node and determine if they reside in the appropriate collection.

Step 2 – Enable Interlock

If you have not already done so, then navigate in UCP to adminadmin settingsLayer 7 Routing.  Then select the check box to Enable Layer 7 Routing.

Step 3 – Label Ingress Nodes

Add an additional label to each node whose purpose is dedicated to running interlock proxies.

$ docker node update --label-add com.docker.interlock.service.cluster=dev ip-127-13-5-3.us-west-2.compute.internal

Repeat this command for each node you are dedicating for ingress traffic and assign appropriate environment values for test and prod.

Step 4 – Download Interlock Configuration

Now that Interlock is running, let us extract its running configuration and modify that to suit our purposes.

CURRENT_CONFIG_NAME=$(docker service inspect --format '{{ (index .Spec.TaskTemplate.ContainerSpec.Configs 0).ConfigName }}' ucp-interlock)
docker config inspect --format '{{ printf "%s" .Spec.Data }}' $CURRENT_CONFIG_NAME > config.orig.toml

The default configuration for Interlock is to have two interlock proxies running anywhere in the cluster. The proxies configuration resides in a section named Extensions.default. This is the heart of an interlock service cluster. We will duplicate this section two times for a total of three sections and then rename them to suit our needs.


Step 5 – Edit Interlock Configuration

Copy the config.orig.toml file to config.new.toml. Then, using your favorite editor (vi of course) duplicate the Extensions.default section two more times.  Rename each of the three Extension.defaults to Extension.dev, Extensions.test, and Extensions.prod.  Each Extensions.<env> section has other sub-sections that include the same name plus a qualifier (e.g. Extensions.default.Config).  These too will need to be renamed.

Now we have 3 named extensions for each of dev, test, and prod.  Next, you will search for the PublishedSSLPort and change it to 8445 for dev, and 8444 for test and leave the value 8443 for prod.  These 3 ports should be the values that the incoming load balancer uses in its back-end pools. For each environment specific VIP (dev, test, prod) the traffic will flow into the load balancer on port 443.  The VIP used to access the load balancer will dictate how the traffic will be routed to the appropriate interlock proxy IP address and port.  

Add a new property called ServiceCluster under each of the extensions sections and give it the name of dev, test, or prod.

You can also specify the constraint labels that will dictate where both the Interlock Extension and Interlock Proxies will run. Start by changing the Constraints and ProxyConstraints to use your new node labels.

The ProxyReplicas indicates how many container replicas to run for the interlock proxy service. We will set ours to 2. The ProxyServiceName is the name of the service as it is deployed into Swarm for this service. We will name ours ucp-interlock-proxy-dev which is specific to the environment it is supporting.

Of course you will do this for all three sections within the new configuration file. Below is a snippet of only the changes that I have made for the dev ingress configuration. You will want to repeat this for test and prod as well.

    ServiceCluster = "dev"
    ProxyServiceName = "ucp-interlock-proxy-dev"
    ProxyReplicas = 1
    PublishedPort = 8082
    PublishedSSLPort = 8445
    Constraints = ["node.labels.com.docker.interlock.service.cluster=dev"]
    ProxyConstraints = ["node.labels.com.docker.interlock.service.cluster=dev"]

This snipped of the configuration file only represents the values that have changed. There are numerous others you should just leave as is.

Step 6 – Upload new Interlock Configuration

NEW_CONFIG_NAME="com.docker.ucp.interlock.conf-$(( $(cut -d '-' -f 2 <<< "$CURRENT_CONFIG_NAME") + 1 ))"
docker config create $NEW_CONFIG_NAME config.new.toml

Step 7- Restart Interlock Service with New Configuration

docker service update --update-failure-action rollback \
    --config-rm $CURRENT_CONFIG_NAME \
    --config-add source=$NEW_CONFIG_NAME,target=/config.toml \
overall progress: 1 out of 1 tasks 
1/1: running   [==================================================>] 
verify: Service converged 

Note: in the above scenario the service update worked smoothly. Other times, such as when there are errors in your configuration, the service will rollback. In those cases you will want to do a docker ps -a | grep interlock and look for the recently exited docker/ucp-interlock container. Once you have its container id you can perform a docker logs <container-id> to see what went wrong.

Step 8 – Verify Everything is Working

We need to make sure that everything started up properly and are listening on their appropriate ports.

docker service ls
ID                  NAME                           MODE                REPLICAS            IMAGE                                  PORTS
y3jg0mka0w7b        ucp-agent                      global              4/5                 docker/ucp-agent:3.1.9                 
xdf9q5y4dev4        ucp-agent-win                  global              0/0                 docker/ucp-agent-win:3.1.9             
k0vb1yloiaqu        ucp-auth-api                   global              0/1                 docker/ucp-auth:3.1.9                  
ki8qeixu12d4        ucp-auth-worker                global              0/1                 docker/ucp-auth:3.1.9                  
nyr40a0zitbt        ucp-interlock                  replicated          0/1                 docker/ucp-interlock:3.1.9             
ewwzlj198zc2        ucp-interlock-extension        replicated          1/1                 docker/ucp-interlock-extension:3.1.9   
yg07hhjap775        ucp-interlock-extension-dev    replicated          1/1                 docker/ucp-interlock-extension:3.1.9   
ifqzrt3kw95p        ucp-interlock-extension-prod   replicated          1/1                 docker/ucp-interlock-extension:3.1.9   
l6zg39sva9bb        ucp-interlock-extension-test   replicated          1/1                 docker/ucp-interlock-extension:3.1.9   
xkhrafdy3czt        ucp-interlock-proxy-dev        replicated          1/1                 docker/ucp-interlock-proxy:3.1.9       *:8082->80/tcp, *:8445->443/tcp
wpelftw9q9co        ucp-interlock-proxy-prod       replicated          1/1                 docker/ucp-interlock-proxy:3.1.9       *:8080->80/tcp, *:8443->443/tcp
g23ahtsxiktx        ucp-interlock-proxy-test       replicated          1/1                 docker/ucp-interlock-proxy:3.1.9       *:8081->80/tcp, *:8444->443/tcp

You can see there are 3 new ucp-interlock-extension-<env> containers and 3 new ucp-interlock-proxy-<env> containers. You can also verify that they are listening on SSL port 8443 thru 8445.  This is fine for a demonstration, but you will more than likely want to set the replica’s somewhere in the 2 to 5 range per environment.  And of course you will determine that based on your traffic load.

NOTE: Often times after the update of the Interlock’s configuration you will still see the old ucp-interlock-extension and/or the ucp-interlock-proxy services still running. You can run the following command to remove these as they are no longer necessary.

docker service rm ucp-interlock-extension ucp-interlock-proxy

Step 9 – Deploy an Application

Now let’s deploy a demo service that we can route thru our new ingress. We’re going to take the standard docker demo application and deploy it to our dev cluster. Start by creating the following docker-compose.yml file:

version: "3.7"
    image: ehazlett/docker-demo
      replicas: 2
        com.docker.lb.hosts: ingress-demo.lab.capstonec.net
        com.docker.lb.network: ingress_dev
        com.docker.lb.port: 8080
        com.docker.lb.service_cluster: dev
        com.docker.ucp.access.label: /dev
      - ingress_dev

    external: true

Note that the com.docker.lb.network attribute is set to ingress_dev. I previously created this network outside of the stack. We will now utilize this network for all our ingress traffic from Interlock to our docker-demo container.

$ docker network create ingress_dev --label com.docker.ucp.access.label=/dev --driver overlay

Also notice that the com.docker.lb.hosts attribute is set to ingress-demo.lab.capstonec.net. I logged into our DNS server and create a CNAME record with that name pointing to my AWS load balancer for the dev environment.

I also must configure my AWS load balancer to allow traffic to a Target Group of virtual machines. We can talk about your cloud configuration in another article down the road.

Let’s deploy that stack:

docker stack deploy -c docker-stack.yml demodev

Once the stack is deployed, we can verify that the services are running on the correct machine:

docker stack ps demodev
ID                  NAME                     IMAGE                         NODE                            DESIRED STATE       CURRENT STATE           ERROR               PORTS
i3bght0p5d0j        demodev_demo.1       ehazlett/docker-demo:latest   ip-127-13-5-3.ec2.internal   Running             Running 10 hours ago                        
cyqfu0ormnn8        demodev_demo.2       ehazlett/docker-demo:latest   ip-127-13-5-3.ec2.internal   Running             Running 10 hours ago                        

Finally we should be able to open a browser to http://ingress-demo.lab.capstonec.net which routes thru the dev interlock service cluster) and see the application running.

You can also run a curl command to verify:

curl ingress-demo.lab.capstonec.net
<!DOCTYPE html>
<html lang="en">
        <meta charset="utf-8">


Well that was a decent amount of work but now you’re done. You’ve successfully implemented your first interlock service cluster which is highly available and segmented into three environments for dev, test, and prod!

As always if you have any questions or need any help please contact us.

Mark Miller
Solutions Architect
Docker Accredited Consultant

What is Container Orchestration?

Over the last two or three years I’ve given a similar presentation on containers to operations groups at clients, potential clients, conferences and meetups. Generally, they’re just getting started with containers and are wondering what orchestration is and how it impacts them. In this post, I will talk about what container orchestration is and provide several videos with simple examples of what it means.

How to securely deploy Docker EE on the AWS Cloud


This reference deployment guide provides the step-by-step instructions for deploying Docker Enterprise Edition on the Amazon Web Services (AWS) Cloud. This automation references deployments that use the Docker Certified Infrastructure (DCI) template which is based on Terraform to launch, configure and run the AWS compute, network, storage and other services required to deploy a specific workload on AWS. The DCI template uses Ansible playbooks to configured the Docker Enterprise cluster environment.

The Power in a Name

My full name is Mark Allen Miller. You can find my profile on LinkedIn under my full name https://www.linkedin.com/in/markallenmiller/. I went to college with two other Mark Millers. One of them also had the same middle initial as me so my name is not the most unique name in the world. My dad’s name is Siegfried Miller. At the age of 18, because he could “change the world”, he changed his last name from Mueller to Miller and yep, he doesn’t have a middle name. My grandfather’s name is Karl Mueller. His Austrian surname, prior to immigrating to the US in 1950, was Müller with an umlaut which is a mark ( ¨ ) used over a vowel to indicate a different vowel quality. Interesting trivia you might say, but what does this have to do with Docker?

Well, Docker originally had the name dotCloud. According to wikipedia “Docker represents an evolution of dotCloud’s proprietary technology, which is itself built on earlier open-source projects such as Cloudlets.” I had never even heard of Cloudlets until I wrote this blog.

Docker containers have names also. These names give us humans something a little more interesting to work with instead of the typical container id such as 648f7f486b24. The name of a container can be used to identify a running instance of an image, but it can also be used in most commands in place of the container id.

Deploying a Docker stack file as a Kubernetes workload


Recently I’ve been hosting workshops for a customer who is exploring migrating from Docker Swarm orchestration to Kubernetes orchestration. The customer is currently using Docker EE (Enterprise Edition) 2.1, and plans to continue using that platform, just leveraging Kubernetes rather than Swarm. There are a number of advantages to continuing to use Docker EE including:

  • Pre-installed Kubernetes.
  • Group (team) and user management, including corporate LDAP integration.
  • Using the Docker UCP client bundle to configure both your Kubernetes and Docker client environment.
  • Availability of an on-premises registry (DTR) that includes advanced features such image scanning and image promotion.

I had already conducted a workshop on deploying applications as Docker services in stack files (compose files deployed as Docker stacks), demonstrating self-healing replicated applications, service discovery and the ability to publish ports externally using the Docker ingress network. …

Viewing Container Logs thru Docker UCP

The Docker Universal Control Plane provides a wealth of information about the Docker cluster. There is information for both Swarm and Kubernetes. There are tons of detailed information about stacks, services, containers, networks, volumes, pods, namespaces, service accounts, controllers, load balancers, pods, configurations, storage, etc. (I think you get the point).

UCP Dashboard

Interlock Service Clusters

The Single-Cluster architecture utilizes a single Docker Swarm cluster with multiple collections to separate the dev, test, and prod worker machines and combined with RBAC it enforces work load isolation of applications across the various runtime environments. Applications deployed to this Single-Cluster can utilize the Interlock reverse proxy capabilities of SSL termination and path based routing. This single Interlock application supports all three collections and the routing of application traffic.

In this article I will show you how to configure Interlock to run in a multi-service-cluster configuration which gains you isolation and dedication of Interlock Proxy instances to each of the dev, test, and prod collections.