Docker Layer 7 Routing – Host Mode

/ Docker, Swarm / / container, docker, interlock, layer 7, routing mesh, ssl termination

It’s been nearly 3 months since my last blog about new the Layer 7 Routing (aka Interlock) in Docker Enterprise 2.0. It’s been a journey of up’s and down’s to get this to work, scale, and become stable enough for a production environment. I’m not sure we can declare total success just yet.

Near the end of my previous blog post I mentioned that there is an alternative configuration for Interlock regarding overlay networks. You could utilize Interlock’s host mode networking. Docker states the following:

By default layer 7 routing leverages the Docker Swarm routing mesh, but you don’t have to. You can use host mode networking for maximum performance.


This is a very technical and detailed article. There are lots of CLI for configuring host mode and deploying your application stack in host mode. The reader could easily turn this CLI into a bash script and automate everything; including the addition of plenty of error checking.

Why use Host Mode?

Remember that the out-of-the-box configuration of Docker’s Interlock is to utilize overlay networks to communicate with upstream applications. This means that the ucp-interlock-proxy service will have to connect to dozen’s upon dozen’s, if not even hundreds, of overlay networks in a large scale enterprise.

It turns out this becomes a convergence issue for ucp-interlock-proxy causing it to take anywhere from 2 to 11 minutes to attach to all these overlay networks. Let’s hope you don’t have too many application deployments occurring or expect any sort of quick response from any particular deployment.

Docker, therefore, recommends using host mode for running interlock services and for communicating to the upstream application services.

Making it Production ready!

Some of the things we wanted to do was come up with a production ready environment. Out of the box, Interlock is not really configured for production. It has a core ucp-interlock service that always runs on a Manager node. It has a ucp-interlock-extension service that runs on any worker node and a ucp-interlock-proxy service that runs two replicas on any worker node.

In our production cluster we have 3 collections: dev, test, prod. Each of these collections have equal number of worker nodes and is easily scaled out by adding new nodes. What we really wanted is for the ucp-interlock-proxy service replicas to run on some dedicated nodes other than our worker nodes.

Configure Interlock for Specific Nodes

In our environment we have 5 DTR nodes which we decided to pin our ucp-interlock-proxy replicas to. Our DTR nodes all have a label called role with a value of dtr which we will leverage.  Docker documentation explains the details.

We start this process by

  • pulling down Interlock’s “configuration” object,
  • modifying it, and
  • create a new configuration with a new name
CURRENT_CONFIG_NAME=$(docker service inspect --format '{{ (index .Spec.TaskTemplate.ContainerSpec.Configs 0).ConfigName }}' ucp-interlock)
docker config inspect --format '{{ printf "%s" .Spec.Data }}' $CURRENT_CONFIG_NAME > config.orig.toml

cat config.orig.toml | \
    sed 's/ Constraints = .*$/ Constraints = \[\"node.labels.role==manager\"\]/g' | \
    sed 's/ProxyConstraints = \[\"node.labels.com.docker.ucp.orchestrator.swarm==true\", \"node.platform.os==linux\"/\ProxyConstraints = \[\"node.labels.role==dtr\"/g' \
    sed -i ’s/ProxyReplicas = 2/ProxyReplicas = 5/g’ \
> config.new.toml

NEW_CONFIG_NAME="com.docker.ucp.interlock.conf-$(( $(cut -d '-' -f 2 <<< "$CURRENT_CONFIG_NAME") + 1 ))" 
docker config create $NEW_CONFIG_NAME config.new.toml

We can then update the interlock service with the new configuration.

docker service update --update-failure-action rollback \
    --config-rm $CURRENT_CONFIG_NAME \
    --config-add source=$NEW_CONFIG_NAME,target=/config.toml \
    ucp-interlock

Notice that we also specified a rollback action in case of any failures. 
Due to a known bug, we must force Interlock Proxy to be constrained to the 5 DTR nodes.

docker service update --constraint-add "node.labels.role==dtr" \
    --replicas=5 ucp-interlock-proxy

Now that the Interlock Proxy service is running on our dedicated DTR nodes, we configure our upstream load balancer with the IP addresses of those nodes. This makes sure all inbound traffic is directed only to these node.

Configure Interlock for Host Mode

I have spent considerable time getting this to work just right and will not bore you with a long history. Docker documentation contains details. We follow the same process as before to change the configuration object.  The attributes being changed are the PublishMode, PublishedPort, and PublishedSSLPort are all being modified within the configuration toml file. These parameters are documented here: https://docs.docker.com/ee/ucp/interlock/deploy/configuration-reference/

CURRENT_CONFIG_NAME2=$(docker service inspect --format '{{ (index .Spec.TaskTemplate.ContainerSpec.Configs 0).ConfigName }}' ucp-interlock) 
docker config inspect --format '{{ printf "%s" .Spec.Data }}' $CURRENT_CONFIG_NAME2> config.orig2.toml

cat config.orig2.toml | \
    sed 's/PublishMode = \"ingress\"/PublishMode = \"host\"/g' | \
    sed 's/PublishedPort = [[:digit:]]*$/PublishedPort = 7080/g' | \
    sed 's/PublishedSSLPort = [[:digit:]]*$/PublishedSSLPort = 7443/g' \
    > config.new2.toml

NEW_CONFIG_NAME2="com.docker.ucp.interlock.conf-$(( $(cut -d '-' -f 2 <<< "$CURRENT_CONFIG_NAME2") + 1 ))"
docker config create $NEW_CONFIG_NAME2 config.new2.toml

Now update the Interlock service with the new configuration. Notice how we are publishing port 8080 on the host.

docker service update --config-rm $CURRENT_CONFIG_NAME2 \
    --config-add source=$NEW_CONFIG_NAME2,target=/config.toml \
    --publish-add mode=host,target=8080 --detach=false ucp-interlock

Stack Deplyment in Host Mode

Ok, so we have our infrastructure all configured using host mode. But we need to deploy our applications using docker stack deploy in a way that utilizes this host mode capability, instead of overlay networks.

  demo:
    image: ehazlett/docker-demo
    environment:
      METADATA: demo
    deploy:
      replicas: 2
      labels:
        - com.docker.ucp.access.label=/dev/ops
        - com.docker.lb.hosts=myapp.acme.com
        - com.docker.lb.port=8080
    ports:
      - target: 8080
        mode: host

The Docker documentation clearly utilizes docker service create in its example. It works fine. But getting this to work in a compose (stack) file is a bit tricky.

Notice that we no longer specify a com.docker.lb.network label. We don’t want the interlock proxy to attempt to connect our network. Also, notice how we have a ports stanza. The target port of 8080 is what the container is listening to internally on the host (not an overlay network). This should be it, but alas … sigh!

Problems

The previous excerpt from a stack file is not enough.

Problem 1

Any declared networks within the stack file are automatically attached to the ucp-interlock-proxy. So if I have 3 backend overlay networks, they all get attached to ucp-interlock-proxy, regardless of whether they are specified in the com.docker.lb.network label.

There currently is not work around for this. We have to wait for Docker to fix this issue since it will exasperate the known container to network convergence issue.

Problem 2

Even with no overlay networks specified anywhere in the stack file, one will be provided with the name of -default. And this too gets attached to the ucp-interlock-proxy even if we don’t specify it in the com.docker.lb.network label.

There is a workaround for this issue; sort of. If you create a network as external and a name of bridge and attach that network to your service, then the default network will not be created.

The trouble with this is that the “name:” field in the stack file is not supported until version 3.5 of the compose api. You must have a docker client of version 17.12.0+ or higher to use this compose version.

Review Solution

I have installed Docker engine 18.09 on my client while my swarm is running 17.06.2-ee-17.

The docker stack deploy command actually performs client side parsing of the stack file. Each of the networks, secrets, volumes, services, etc. are submitted to the swarm one at a time.

Let’s review the final solution that has some work arounds for known problems.

version: "3.5"
services:
  demo:
    image: ehazlett/docker-demo
    environment:
      METADATA: demo
    deploy:
      replicas: 2
      labels:
        - com.docker.ucp.access.label=/dev/ops
        - com.docker.lb.hosts=myapp.acme.com
        - com.docker.lb.port=8080
    ports:
      - target: 8080
        mode: host
    networks:
      - pleaseusethedefaultdocker0bridge

networks:
  pleaseusethedefaultdocker0bridge:
    external: true
    name: bridge

More Problems

During this discovery process we encountered a couple other issues.

Problem 3

When a user deploys an application utilizing Interlock, it eventually causes a rolling update of the ucp-interlock-proxy service replicas. We noticed this was occuring on every stack deployment, even when the stack did not specify Interlock.

We reported the problem to Docker who acknowledged it was a known issue and that they were working it. UCP patch 3.0.6 fixed this issue.

Problem 4

At another point in time, I had deployed a stack file which had a syntactical error in the com.docker.lb.hosts label. This made it all the way into the nginx.conf file that feeds ucp-interlock-proxy. This caused the proxy containers to redeploy but as they started up, they failed to read the nginx.conf file. So they kept on flapping: up – down – up – down.

At no point in time should a developer’s typo be able to take out a production service like Interlock that is handling all inbound traffic to the swarm. There is an option to have the ucp-interlock-proxy service rollback upon failure which is only a temporary workaround. While the workaround will keep the Interlock services up and running, it does not notify any users and it does not reject the bad data in the first place.

Docker was also notified of this issue and is addressing it.

Outstanding Problems

Problem 1 and 4 are truly outstanding issues.

Problem 1 – The fact that Interlock is attaching itself to all of the overlay networks in your stack just exasperates the Interlock convergence issue (which is the very reason we are trying to goto host mode in the first pace).

Problem 4 – Docker really should perform some level of validation on the values of the com.docker.lb.* labels. If any of these labels has even a single character that ucp-interlock-proxy (nginx) cannot handle, then the proxy containers start flapping.

The Future

In the near future I plan on investigating the use of service clusters to compartmentalize three different interlock services; one for each collection within my cluster.

Conclusion

Layer 7 Routing (Interlock) with host mode should solve the Interlock convergence issue, but falls short. The Interlock product is very promising. It’s SSL termination and path based routing are a welcome features beyond what HRM provided in a prior release. However, we will await fixes from Docker before we can consider this ready for prime-time use.