Evaluation of post mortem forensics for containerd

This gist evalutes how one can setup and use the new tool container-explorer to perform post mortem forensics using an image of a host that contained a containerd installation.

The following steps are performed on a Windows WLS2 shell running an Ubuntu image. Therefore the steps should also work for native ubuntu.

Mount the evidence image

First, check the file system of your exported image. This can be anything from VirtualBox disk to whatever someone throw at your desk. I am using a raw disk export from VirtualBox. This image is the export of a worker node from a kubernetes cluster. Disclaimer: This is all test data! If you want to know how to setup a Kubernetes cluster in VirtualBox, see here.

fdisk -l /mnt/i/Kubernetes_DeployPod/DeployPod/kubernetes-worker-1_DeployPod.20.raw

Disk /mnt/i/Kubernetes_DeployPod/DeployPod/kubernetes-worker-1_DeployPod.20.raw: 128 GiB, 137438953472 bytes, 268435456 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x3b09fe96

Device                                                                      Boot   Start       End   Sectors   Size Id Type
/mnt/i/Kubernetes_DeployPod/DeployPod/kubernetes-worker-1_DeployPod.20.raw1 *       2048    999423    997376   487M 83 Linux
/mnt/i/Kubernetes_DeployPod/DeployPod/kubernetes-worker-1_DeployPod.20.raw2       999424   4999167   3999744   1.9G 82 Linux swap / Solaris
/mnt/i/Kubernetes_DeployPod/DeployPod/kubernetes-worker-1_DeployPod.20.raw3      4999168 268433407 263434240 125.6G 83 Linux

We want to take a look into the last partition. To calculate the offset, multiply the Sector Size with the Start sector.

512 * 4999168 = 2559574016

Now mount the image.

# create a mount point
sudo mkdir /mnt/investigation

# now mount your partition (you obviously might play around with the options)
sudo mount -t auto -o ro,loop,norecovery,offset=2559574016 /mnt/i/Kubernetes_DeployPod/DeployPod/kubernetes-worker-1_DeployPod.20.raw /mnt/investigation/

# verify (test)
ls -la /mnt/investigation/

total 88
drwxr-xr-x  19 root root  4096 Aug 20 21:44 .
drwxr-xr-x  11 root root  4096 Nov 14 02:16 ..
lrwxrwxrwx   1 root root     7 Aug 20 21:13 bin -> usr/bin
drwxr-xr-x   2 root root  4096 Aug 20 21:13 boot
drwxr-xr-x   4 root root  4096 Aug 20 21:13 dev
drwxr-xr-x 103 root root  4096 Nov 12 01:23 etc
drwxr-xr-x   3 root root  4096 Aug 20 21:20 home
lrwxrwxrwx   1 root root     7 Aug 20 21:13 lib -> usr/lib
lrwxrwxrwx   1 root root     9 Aug 20 21:13 lib32 -> usr/lib32
lrwxrwxrwx   1 root root     9 Aug 20 21:13 lib64 -> usr/lib64
lrwxrwxrwx   1 root root    10 Aug 20 21:13 libx32 -> usr/libx32
drwx------   2 root root 16384 Aug 20 21:13 lost+found
drwxr-xr-x   3 root root  4096 Aug 20 21:13 media
drwxr-xr-x   2 root root  4096 Jul 31  2020 mnt
drwxr-xr-x   4 root root  4096 Nov 12 01:23 opt
drwxr-xr-x   2 root root  4096 Apr 15  2020 proc
drwx------   3 root root  4096 Aug 20 21:31 root
drwxr-xr-x   2 root root  4096 Aug 20 21:21 run
lrwxrwxrwx   1 root root     8 Aug 20 21:13 sbin -> usr/sbin
drwxr-xr-x   2 root root  4096 Aug 20 21:21 snap
drwxr-xr-x   2 root root  4096 Jul 31  2020 srv
drwxr-xr-x   2 root root  4096 Apr 15  2020 sys
drwxrwxrwt  12 root root  4096 Nov 12 11:32 tmp
drwxr-xr-x  15 root root  4096 Aug 20 21:26 usr
drwxr-xr-x  13 root root  4096 Aug 20 21:17 var

Mounted. Noice.

Build and run the tool

Okay my friends, two options: You either may have golang already installed and setup on your system, or you may use docker in order to run the golang container image. I will use the container image in docker. But besides the docker run command all further steps are identical.

# (OPTIONAL) Start the container 
docker run --rm -it -v /mnt/investigation/:/tmp/investigation golang

# in a directory of your choice, clone the repo
root@166117f40b35:/go# git clone https://github.com/google/container-explorer.git

Cloning into 'container-explorer'...
remote: Enumerating objects: 18, done.
remote: Counting objects: 100% (18/18), done.
remote: Compressing objects: 100% (16/16), done.
remote: Total 18 (delta 1), reused 18 (delta 1), pack-reused 0
Receiving objects: 100% (18/18), 53.93 KiB | 2.07 MiB/s, done.
Resolving deltas: 100% (1/1), done.

# change directory
cd container-explorer/

# build the project
go build .

# copy the resulting binary to /bin/
cp ./container-explorer /bin

# validate
container-explorer --help

NAME:
   container-explorer - standalone container explorer

USAGE:
   container-explorer [global options] command [command options] [arguments...]

VERSION:
   0.0.1

DESCRIPTION:
   standalone container explorer

COMMANDS:
   list, ls   list containerd information
   mount      mount container
   mount-all  mount all containers
   help, h    Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --debug                                   enable debug messages
   --container-root value, -c value          containerd root directory (default: "/var/lib/containerd")
   --image-root value, -i value              mounted virtual machine image root directory
   --manifest-file value, -m value           containerd manifest meta.db
   --snapshot-metadata-file value, -s value  containerd snapshot metadata database metadata.db
   --namespace value, -n value               containerd namespace required with mount command (default: "default")
   --help, -h                                show help
   --version, -v                             print the version

Now as the binary has been succesfully built, use it! :)

Try to list containers

container-explorer --image-root /tmp/investigation list containers

NAMESPACE       CONTAINER NAME                                                          IMAGE                                           CREATED AT                      LABELS
k8s.io          0543db13fd21e959a6dd8f28ea2dd44c271cf77d959b123a08790534b08f79dd        k8s.gcr.io/pause:3.2                            2021-11-12T00:23:56.532330089Z  io.kubernetes.pod.name=kube-proxy-x826s,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.uid=bc227b1a-af55-4f2f-ba30-97285a532981,k8s-app=kube-proxy,pod-template-generation=1,controller-revision-hash=674d79d6f9,io.cri-containerd.kind=sandbox
k8s.io          1858c2b2a9e0672cced55ef7d5c5aed9a5f14c50eccff67882e2618fbdc9f02e        docker.io/calico/node:v3.15.1                   2021-11-12T00:24:42.952621043Z  io.cri-containerd.kind=container,io.kubernetes.container.name=calico-node,io.kubernetes.pod.name=calico-node-rvkh4,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.uid=ae2e2cc4-7c8e-483e-b834-413e9fd6fb5c
k8s.io          430238aaeaeddcd84b69f5f1d217c727a332d4c5bbc755f00e37831f6fc2e149        k8s.gcr.io/kube-proxy:v1.22.3                   2021-11-12T00:24:18.965026691Z  io.cri-containerd.kind=container,io.kubernetes.container.name=kube-proxy,io.kubernetes.pod.name=kube-proxy-x826s,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.uid=bc227b1a-af55-4f2f-ba30-97285a532981
k8s.io          53ecc572b24eca56b957029d0e5be381be1b6f469ea8674718019ff1b70c16ca        docker.io/library/busybox:latest                2021-11-12T10:32:07.960108316Z  io.kubernetes.pod.namespace=default,io.kubernetes.pod.uid=564f434f-cb55-4385-b874-33fe31ff31db,io.cri-containerd.kind=container,io.kubernetes.container.name=simple-pod-container,io.kubernetes.pod.name=simple-pod
k8s.io          8541f3c0b5bfeddd0d2495a0f297e05530757a718021e73151b4d6a3bd39a50e        k8s.gcr.io/pause:3.2                            2021-11-12T10:32:03.601601770Z  io.kubernetes.pod.name=simple-pod,io.kubernetes.pod.namespace=default,io.kubernetes.pod.uid=564f434f-cb55-4385-b874-33fe31ff31db,io.cri-containerd.kind=sandbox
k8s.io          b5634a33912e523893a15d71b9080e1c36ee408f8bf565a2b3b06f2f093303f0        docker.io/calico/pod2daemon-flexvol:v3.15.1     2021-11-12T00:24:28.755497046Z  io.cri-containerd.kind=container,io.kubernetes.container.name=flexvol-driver,io.kubernetes.pod.name=calico-node-rvkh4,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.uid=ae2e2cc4-7c8e-483e-b834-413e9fd6fb5c
k8s.io          beefe9a39a34ce320e3949ab9edc446bbb1f4b26d7b263dc5de4fa1dab0f5070        k8s.gcr.io/pause:3.2                            2021-11-12T00:23:56.522271623Z  io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.uid=ae2e2cc4-7c8e-483e-b834-413e9fd6fb5c,k8s-app=calico-node,pod-template-generation=1,controller-revision-hash=68b8fd8c6d,io.cri-containerd.kind=sandbox,io.kubernetes.pod.name=calico-node-rvkh4
k8s.io          bf795d6c04c80dcad92bfb8d1894b76c24c58c7d18a68bc8ed74eb02aac3593c        docker.io/calico/cni:v3.15.1                    2021-11-12T00:24:10.865995409Z  io.kubernetes.pod.name=calico-node-rvkh4,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.uid=ae2e2cc4-7c8e-483e-b834-413e9fd6fb5c,io.cri-containerd.kind=container,io.kubernetes.container.name=upgrade-ipam
k8s.io          f1c275c0ff2d2a02faa8f83e8cdbfb1756301d10f46ca38f2ca24f63619cbce1        docker.io/calico/cni:v3.15.1                    2021-11-12T00:24:12.894172300Z  io.kubernetes.container.name=install-cni,io.kubernetes.pod.name=calico-node-rvkh4,io.kubernetes.pod.namespace=kube-system,io.kubernetes.pod.uid=ae2e2cc4-7c8e-483e-b834-413e9fd6fb5c,io.cri-containerd.kind=container

Or list the snapshots:

container-explorer --image-root /tmp/investigation list snapshot

<output redacted>

Follow the official repo for further commands and docs.