Why I'm Not Using Docker Secrets with Docker Compose

---

I was looking for a relatively simple and portable method of handling secrets (i.e. confidential data like usernames and passwords) in my projects. Something that demonstrated the principles and practices of (good) secret management that was still useable locally, in Continuous Integration/Continuous Deployment (CI/CD), and in production.

From the application perspective, I wanted a general mechanism for secret handling and retrieval that could be used regardless of the external secret management system. Something that could be used when running locally natively, locally in Docker Compose, and in a potential Kubernetes production deployment. A pattern that could be used in any application in any language.

Since I was already using Docker Compose for my projects both locally and in my CI using GitHub Actions. I thought that I would start with trying Docker secrets.

This is my experience and learnings during this failed journey.

---

Overview of Docker Secrets

Distributed Centralized Management, Encrypted in Transit and Rest

Docker secrets offers centralized management with distributed replication of confidential data that is encrypted both in transit and at rest. Which is great.

Mounted as File in Container

Through configuration, a docker secret is "mounted" into the running container as an (encrypted and in-memory) file. This requires that your application can retrieve (read) individual secrets from a specified file. Docker secrets are mounted into the container at /run/secrets/<secret_name> (for Linux containers). There is by intent no Docker support for using environment variables to retrieve Docker secrets as environment variables can "leak" between Docker containers (e.g. --link).

The general convention is to specify the secret file location to the application by environment variable with a _FILE suffix (e.g. MYSQL_ROOT_PASSWORD_FILE).

If your application is already designed to retrieve a secret from an environment variable, the recommended approach by Docker is to maintain the environment variable specification (e.g. MYSQL_ROOT_PASSWORD) for backwards compatibility but add an additional environment variable to supply the secret file (e.g. MYSQL_ROOT_PASSWORD_FILE).

This is all a pretty good approach and is reasonably portable.

---

Only Fully Supported in Docker Swarm

Docker secrets were originally implemented for and only supported by Docker Swarm - Docker's container orchestration tool (think Kubernetes).

🔬 Docker secrets are stored in the Swarm manager's encrypted Raft log which is replicated on all Swarm managers for high availability.

Although (limited) support was later added for using Docker secrets with Docker Compose, it only supports creating Docker secrets from local files. To use external Docker secrets using the docker secret create command, you must use Docker Swarm and execute the docker secret create command on a Swarm manager.

This limitation was a deal breaker.

I wanted a secrets solution for Docker Compose both locally and in my CI and I saw no security in Docker secrets with accessible source text files on the Docker Compose host.

That said, here is what I learned with some code and examples of running the commands.

---

My Journey with Docker Secrets in Docker Compose

My plan was to use external Docker secrets in my Docker Compose framework. Locally, I would have a simple (but totally unsecure) script to create the Docker secrets (using the docker secret create command) which would then be used by my project's Docker Compose framework when I launched it with docker-compose run. Then in my CI with GitHub Actions, I would use my simple script to create the Docker secrets and run the Docker Compose framework for the project's functional testing. Another approach for my CI would have been to store the secret values in GitHub Secrets and use the docker secret create command. But this was not going to work because external Docker secrets can not be used with Docker Compose.

---

The Exploratory "App"

For my exploration with Docker secrets I needed a simple application to test the accessibility of Docker secrets in the application.

Since the projects in which I was planing to use Docker secrets are in Ruby, I created a very simple Ruby program read_secret_from_file that reads the Docker secrets from the specified files and displays them.

#!/usr/bin/env ruby
# frozen_string_literal: true

# Read the secrets
app_secret_1 = IO.read(ENV['APP_SECRET_1_FILE']).chomp
app_secret_2 = IO.read(ENV['APP_SECRET_2_FILE']).chomp

# Output the secrets
puts "app_secret_1 = [#{app_secret_1}]"
puts "app_secret_2 = [#{app_secret_2}]"

---

The Dockerfile

Now that I had my simple app, I needed a simple Dockerfile to create an image for it. I used Ruby Alpine as it is an official Ruby image and Alpine is small and fast to build.

FROM ruby:2.7.5-alpine
COPY . .
CMD ./read_secret_from_file

I then built the image readsecret using the docker build command...

docker build -t readsecret .

---

The Initial docker-compose File and Secrets from Local Files

I initially started with creating the Docker secrets from local text files specified in the docker-compose file because I was pretty sure that it would work and knew that it would be easier to repeat for debugging and exploring and be more ephemeral. It was also how stackoverflow said to do it without using Docker Swarm.

Here is my initial docker-compose.yml file creating the Docker secrets from local text files at Docker Compose time...

version: '3.1'
services:
  readsecret:
    image: readsecret
    environment:
      APP_SECRET_1_FILE: /run/secrets/app_secret_1
      APP_SECRET_2_FILE: /run/secrets/app_secret_2
    secrets:
      - app_secret_1
      - app_secret_2

secrets:
   app_secret_1:
     file: appsecret1.txt
   app_secret_2:
     file: appsecret2.txt

Here are the local files with the secret values...

1. appsecret1.txt...

   tomsmith
   

2. appsecret2.txt...

   SuperSecretPassword!
   

---

Running Docker Compose with Local File-Based Secrets

When I ran my exploratory container with docker-compose run readsecret...

% docker-compose run readsecret

Creating ruby-docker-secrets_readsecret_run ... done
app_secret_1 = [tomsmith]
app_secret_2 = [SuperSecretPassword!]

Success. You can see that it successfully created the secrets, read them from the Docker secrets files, and displayed their values.

I also proved that if I "shelled" into the container, I could read the Docker secrets files...

% docker-compose run readsecret sh
Creating ruby-docker-secrets_readsecret_run ... done
/ # ls -al /run/secrets/app_secret_1
-rw-r--r--    1 root     root             9 Jan 19 15:32 /run/secrets/app_secret_1
/ # cat /run/secrets/app_secret_1
tomsmith
/ # exit

At this point, I knew that my app, image, and initial docker-compose file with local file-based Docker secrets worked, I was then ready to try using the external Docker secret configuration that I really wanted.

---

The Desired docker-compose File Using External Docker Secrets

Instead of using the local text files and creating my secrets at Docker Compose time (ie. docker-compose run), I wanted to use external Docker secrets created outside of docker-compose run (i.e. external secret management system).

I started with changing my docker-compose.yml file to use external secrets (external: true) under the secrets definition...

version: '3.1'
services:
  readsecret:
    image: readsecret
    environment:
      APP_SECRET_1_FILE: /run/secrets/app_secret_1
      APP_SECRET_2_FILE: /run/secrets/app_secret_2
    secrets:
      - app_secret_1
      - app_secret_2

secrets:
   app_secret_1:
     external: true
   app_secret_2:
     external: true

And being a follower of Test-Driven Development, I wanted to ensure that this would fail since I had not yet created the external secrets...

% docker-compose run readsecret

WARNING: Service "readsecret" uses secret "app_secret_1" which is external. External secrets are not available to containers created by docker-compose.
WARNING: Service "readsecret" uses secret "app_secret_2" which is external. External secrets are not available to containers created by docker-compose.
Creating network "ruby-docker-secrets_default" with the default driver
Creating ruby-docker-secrets_readsecret_run ... done
Traceback (most recent call last):
	1: from ./read_secret_from_file:5:in `<main>'
./read_secret_from_file:5:in `read': No such file or directory @ rb_sysopen - /run/secrets/app_secret_1 (Errno::ENOENT)
ERROR: 1

Uh-oh. Although it failed because it was unable to read the secrets that had not yet been created as I expected, those WARNING messages were the first sign that my plan was not going to work.

But I wanted to continue my exploration and try to create these external secrets to see if they might work.

---

Creating Docker Secrets

Now I needed to create my external Docker secrets.

I tried using the docker secret create command as described in the Docker secret create documentation...

% printf "tomsmith" | docker secret create app_secret_1 -

Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.

That was not good. I had not yet used Swarm, but I tried what the error message said...

% docker swarm init

Swarm initialized: current node (ljg5q5dazv5s5u5nxxeak9l8t) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-1r7ip3t3rvom50tgwmf75722w9gsde73fnjls1zi06r7f8y2px-2t0crx2309lot86b3b6lxn4yh 192.168.65.3:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

I was able initialize current node as a manager.

Now I was able to create the first Docker secret...

printf "tomsmith" | docker secret create app_secret_1 -
ktdmea6tmwynp04vw7jbz1hqo

I wanted to prove that I could inspect the secret...

% docker secret inspect app_secret_1

[
    {
        "ID": "ktdmea6tmwynp04vw7jbz1hqo",
        "Version": {
            "Index": 11
        },
        "CreatedAt": "2022-01-20T21:17:31.7442047Z",
        "UpdatedAt": "2022-01-20T21:17:31.7442047Z",
        "Spec": {
            "Name": "app_secret_1",
            "Labels": {}
        }
    }
]

---

Running Docker Compose with External Docker Secrets (Fail)

Now I tried running my app with docker-compose run hoping it would at least read the first secret and then fail on the second...

% docker-compose run readsecret

WARNING: Service "readsecret" uses secret "app_secret_1" which is external. External secrets are not available to containers created by docker-compose.
WARNING: Service "readsecret" uses secret "app_secret_2" which is external. External secrets are not available to containers created by docker-compose.
Creating ruby-docker-secrets_readsecret_run ... done
Traceback (most recent call last):
	1: from ./read_secret_from_file:5:in `<main>'
./read_secret_from_file:5:in `read': No such file or directory @ rb_sysopen - /run/secrets/app_secret_1 (Errno::ENOENT)
ERROR: 1

Fail. It still gave me the same warnings about "[e]xternal secrets are not available to containers created by Docker Compose" and it still failed in my app because it was unable to read the external Docker secret.

My internet searches also failed to show an example using external Docker secrets and all mentioned the local file configuration for Docker Compose.

It was worth a shot. If the local file approach works for your needs or if you are already using Docker Swarm, then Docker secrets could work great for you.

---