I was looking for a relatively simple and portable method of handling secrets (i.e. confidential data like usernames and passwords) in my projects. Something that demonstrated the principles and practices of (good) secret management that was still useable locally, in Continuous Integration/Continuous Deployment (CI/CD), and in production.
From the application perspective, I wanted a general mechanism for secret handling and retrieval that could be used regardless of the external secret management system. Something that could be used when running locally natively, locally in Docker Compose, and in a potential Kubernetes production deployment. A pattern that could be used in any application in any language.
Since I was already using Docker Compose for my projects both locally and in my CI using GitHub Actions. I thought that I would start with trying Docker secrets.
This is my experience and learnings during this failed journey.
Docker secrets offers centralized management with distributed replication of confidential data that is encrypted both in transit and at rest. Which is great.
Through configuration, a docker secret is "mounted" into the running
container as an (encrypted and in-memory) file. This requires
that your application can retrieve (read) individual secrets from a
specified file. Docker secrets are mounted into the container at
/run/secrets/<secret_name> (for Linux containers). There is by
intent no Docker support for using environment variables to retrieve
Docker secrets as environment variables can "leak" between Docker
containers (e.g. --link).
The general convention is to specify the secret file location to the
application by environment variable with a _FILE suffix
(e.g. MYSQL_ROOT_PASSWORD_FILE).
If your application is already designed to retrieve a secret from
an environment variable, the recommended approach by Docker is to
maintain the environment variable specification
(e.g. MYSQL_ROOT_PASSWORD) for backwards compatibility but add
an additional environment variable to supply the secret file
(e.g. MYSQL_ROOT_PASSWORD_FILE).
This is all a pretty good approach and is reasonably portable.
Docker secrets were originally implemented for and only supported by Docker Swarm - Docker's container orchestration tool (think Kubernetes).
🔬 Docker secrets are stored in the Swarm manager's encrypted Raft log which is replicated on all Swarm managers for high availability.
Although (limited) support was later
added
for using Docker secrets with Docker Compose, it only
supports creating Docker secrets from local files. To use
external Docker secrets using the docker secret create
command, you must use Docker Swarm and execute the docker secret create command
on a Swarm manager.
This limitation was a deal breaker.
I wanted a secrets solution for Docker Compose both locally and in my CI and I saw no security in Docker secrets with accessible source text files on the Docker Compose host.
That said, here is what I learned with some code and examples of running the commands.
My plan was to use external Docker secrets in my Docker Compose
framework. Locally, I would have a simple (but totally unsecure)
script to create the Docker secrets (using the docker secret create
command) which would then be used by my project's Docker Compose
framework when I launched it with docker-compose run. Then in
my CI with GitHub Actions, I would use my simple script to create
the Docker secrets and run the Docker Compose framework for the
project's functional testing. Another approach for my CI would have
been to store the secret values in GitHub Secrets and use the
docker secret create command. But this was not going to work
because external Docker secrets can not be used with Docker Compose.
For my exploration with Docker secrets I needed a simple application to test the accessibility of Docker secrets in the application.
Since the projects in which I was planing to use Docker secrets
are in Ruby, I created a very simple Ruby program
read_secret_from_file that reads the Docker secrets from the
specified files and displays them.
#!/usr/bin/env ruby
# frozen_string_literal: true
# Read the secrets
app_secret_1 = IO.read(ENV['APP_SECRET_1_FILE']).chomp
app_secret_2 = IO.read(ENV['APP_SECRET_2_FILE']).chomp
# Output the secrets
puts "app_secret_1 = [#{app_secret_1}]"
puts "app_secret_2 = [#{app_secret_2}]"Now that I had my simple app, I needed a simple Dockerfile to create
an image for it. I used Ruby Alpine
as it is an official Ruby image and Alpine is small and fast to build.
FROM ruby:2.7.5-alpine
COPY . .
CMD ./read_secret_from_fileI then built the image readsecret using the docker build command...
docker build -t readsecret .
I initially started with creating the Docker secrets from local text files specified in the docker-compose file because I was pretty sure that it would work and knew that it would be easier to repeat for debugging and exploring and be more ephemeral. It was also how stackoverflow said to do it without using Docker Swarm.
Here is my initial docker-compose.yml file creating the Docker
secrets from local text files at Docker Compose time...
version: '3.1'
services:
readsecret:
image: readsecret
environment:
APP_SECRET_1_FILE: /run/secrets/app_secret_1
APP_SECRET_2_FILE: /run/secrets/app_secret_2
secrets:
- app_secret_1
- app_secret_2
secrets:
app_secret_1:
file: appsecret1.txt
app_secret_2:
file: appsecret2.txtHere are the local files with the secret values...
-
appsecret1.txt...tomsmith -
appsecret2.txt...SuperSecretPassword!
When I ran my exploratory container with
docker-compose run readsecret...
% docker-compose run readsecret
Creating ruby-docker-secrets_readsecret_run ... done
app_secret_1 = [tomsmith]
app_secret_2 = [SuperSecretPassword!]
Success. You can see that it successfully created the secrets, read them from the Docker secrets files, and displayed their values.
I also proved that if I "shelled" into the container, I could read the Docker secrets files...
% docker-compose run readsecret sh
Creating ruby-docker-secrets_readsecret_run ... done
/ # ls -al /run/secrets/app_secret_1
-rw-r--r-- 1 root root 9 Jan 19 15:32 /run/secrets/app_secret_1
/ # cat /run/secrets/app_secret_1
tomsmith
/ # exit
At this point, I knew that my app, image, and initial docker-compose file with local file-based Docker secrets worked, I was then ready to try using the external Docker secret configuration that I really wanted.
Instead of using the local text files and creating my secrets at
Docker Compose time (ie. docker-compose run), I wanted to use
external Docker secrets created outside of docker-compose run
(i.e. external secret management system).
I started with changing my docker-compose.yml file to use
external secrets (external: true) under the secrets definition...
version: '3.1'
services:
readsecret:
image: readsecret
environment:
APP_SECRET_1_FILE: /run/secrets/app_secret_1
APP_SECRET_2_FILE: /run/secrets/app_secret_2
secrets:
- app_secret_1
- app_secret_2
secrets:
app_secret_1:
external: true
app_secret_2:
external: true
And being a follower of Test-Driven Development, I wanted to ensure that this would fail since I had not yet created the external secrets...
% docker-compose run readsecret
WARNING: Service "readsecret" uses secret "app_secret_1" which is external. External secrets are not available to containers created by docker-compose.
WARNING: Service "readsecret" uses secret "app_secret_2" which is external. External secrets are not available to containers created by docker-compose.
Creating network "ruby-docker-secrets_default" with the default driver
Creating ruby-docker-secrets_readsecret_run ... done
Traceback (most recent call last):
1: from ./read_secret_from_file:5:in `<main>'
./read_secret_from_file:5:in `read': No such file or directory @ rb_sysopen - /run/secrets/app_secret_1 (Errno::ENOENT)
ERROR: 1
Uh-oh. Although it failed because it was unable to read the
secrets that had not yet been created as I expected, those
WARNING messages were the first sign that my plan was not
going to work.
But I wanted to continue my exploration and try to create these external secrets to see if they might work.
Now I needed to create my external Docker secrets.
I tried using the docker secret create command as described in the
Docker secret create documentation...
% printf "tomsmith" | docker secret create app_secret_1 -
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
That was not good. I had not yet used Swarm, but I tried what the error message said...
% docker swarm init
Swarm initialized: current node (ljg5q5dazv5s5u5nxxeak9l8t) is now a manager.
To add a worker to this swarm, run the following command:
docker swarm join --token SWMTKN-1-1r7ip3t3rvom50tgwmf75722w9gsde73fnjls1zi06r7f8y2px-2t0crx2309lot86b3b6lxn4yh 192.168.65.3:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
I was able initialize current node as a manager.
Now I was able to create the first Docker secret...
printf "tomsmith" | docker secret create app_secret_1 -
ktdmea6tmwynp04vw7jbz1hqo
I wanted to prove that I could inspect the secret...
% docker secret inspect app_secret_1
[
{
"ID": "ktdmea6tmwynp04vw7jbz1hqo",
"Version": {
"Index": 11
},
"CreatedAt": "2022-01-20T21:17:31.7442047Z",
"UpdatedAt": "2022-01-20T21:17:31.7442047Z",
"Spec": {
"Name": "app_secret_1",
"Labels": {}
}
}
]
Now I tried running my app with docker-compose run hoping it would
at least read the first secret and then fail on the second...
% docker-compose run readsecret
WARNING: Service "readsecret" uses secret "app_secret_1" which is external. External secrets are not available to containers created by docker-compose.
WARNING: Service "readsecret" uses secret "app_secret_2" which is external. External secrets are not available to containers created by docker-compose.
Creating ruby-docker-secrets_readsecret_run ... done
Traceback (most recent call last):
1: from ./read_secret_from_file:5:in `<main>'
./read_secret_from_file:5:in `read': No such file or directory @ rb_sysopen - /run/secrets/app_secret_1 (Errno::ENOENT)
ERROR: 1
Fail. It still gave me the same warnings about "[e]xternal secrets are not available to containers created by Docker Compose" and it still failed in my app because it was unable to read the external Docker secret.
My internet searches also failed to show an example using external Docker secrets and all mentioned the local file configuration for Docker Compose.
It was worth a shot. If the local file approach works for your needs or if you are already using Docker Swarm, then Docker secrets could work great for you.
Hi @brianjbayer,
Thanks for the detailed write up. I tried external secrets with docker-compose initially, it didn't work. Later moved to Docker Swarm after going through your article, but not able to create the service using the external secrets. Got "unsupported external secret" when used docker stack deploy using docker-compose as input, by inspecting docker stack "docker stack ps --no-trunc" I see a error message "mkdir /var/lib/docker: read-only file system", which is due to insufficient permissions i.e. installation of docker via snap, uninstalled and installed docker from official docker installation docs, it worked this time.