Tech ChroniclesRamblings of a Tech Dude
Hack Your Docker Container

Hack Your Docker Container

docker_security

 

Secure Docker containers are crucial for your mission-critical services. You might be running your containers using sophisticated orchestrators such as Kubernetes (K8S) in Google Cloud Platform (GCP). You might think that you have implemented extremely safe role-based access controls (RBAC) in your environment. You have not even enabled an OS shell in the running containers. You might think that your services are super secure. Still, somebody will be able to take out crucial information out of your Docker containers that can tarnish your reputation. What do you do for that? Before somebody hack your Docker container, you hack your own Docker container. Try to break it with the best of your abilities. Early on.

In this story, you will get to have a look at some of the easy ways in which you can take a peep into your Docker images and containers so that you can make sure that some of the obvious doors are not kept open and intruders are not exploiting those vulnerabilities. This story is structured in such a way that a trivial application is built on Alpine Linux using Python first, and its Docker container is used for further exploration.

A Trivial Application

In this story, as mentioned earlier, for demonstration purposes, a trivial Docker image is built using Python, and Flask running on Alpine Linux. When this Docker container is running, it serves a simple REST endpoint on the root path “/” responding with the message “Hello from Flask”. The reason for choosing Alpine Linux as the OS because that is one of the slimmest Linux distributions used widely for building Docker images. Python is chosen because it is very easy to build services using Python but it is very easy to get exposed as well. The following tree displays the directory structure of the application code base.

$ tree
.
├── Dockerfile
└── app
    ├── main.py
    └── requirements.txt

The requirements.txt file contains the list of Python libraries to be installed in the Docker image. Since this is a very trivial application, there is only one Python library that needs to be installed. See the following content that should be present in the requirements list file that will be used by PIP, the package manager for Python.

Flask==0.10.1

The main.py file that contains the application’s REST endpoint definitions. Take a look at the following contents that should go into main.py.

from flask import Flask
app = Flask(__name__)@app.route('/')
def hello_world():
    return 'Hello from Flask'if __name__ == '__main__':
    app.run(debug=False,host='0.0.0.0')

The most important file that you need to build the Docker image for this application is the Dockerfile. The Docker image is built on top of the base image of Alpine Linux 3.9 with Python 3.7.3. It is ideal that this image not to have the OS shell or any other utilities for anybody to do any kind of script injection or anything of that sort. But for demonstrating some vulnerabilities, a deliberate attempt is made to have the OS shell enabled as that comes as default. Take a look at the following contents that should go into Dockerfile.

FROM python:3.7.3-alpine3.9
COPY ./app /app
WORKDIR /app
RUN pip install -r /app/requirements.txt
ENTRYPOINT ["python"]
CMD ["main.py"]

Now you have all the required source files organized in a well-formed tree structure, you can build the Docker image using the following commands from your local development machine’s terminal window. No need to mention that you should have Docker running in your development machine as a pre-requisite.

$ export CONTAINER_NAME=flask
$ echo $CONTAINER_NAME
$ pwd
/Users/RajT/workspace/Docker/alpine-flask
$ docker build -t $CONTAINER_NAME .
Sending build context to Docker daemon  4.608kB
Step 1/6 : FROM python:3.7.3-alpine3.9
 ---> a93594ce93e7
Step 2/6 : COPY ./app /app
 ---> Using cache
 ---> 14b5b37c636e
Step 3/6 : WORKDIR /app
 ---> Using cache
 ---> 2a7c3ce7ca45
Step 4/6 : RUN pip install -r /app/requirements.txt
 ---> Using cache
 ---> 0a4848444aa9
Step 5/6 : ENTRYPOINT ["python"]
 ---> Using cache
 ---> ace29865523b
Step 6/6 : CMD ["main.py"]
 ---> Using cache
 ---> b3a3bceb4943
Successfully built b3a3bceb4943
Successfully tagged flask:latest
$ docker run --name $CONTAINER_NAME -d -p 5000:5000 $CONTAINER_NAME
012cbcfb6dc10f18f30482c263b1be0a99cb00d017c744ba7d5e662779954833
$ curl localhost:5000/
Hello from Flask

Once you layout your files and follow the commands listed above, you will be able to see that the Docker container is running well and it is serving requests. The environment variable defined in your local development machine CONTAINER_NAME is used to store the Docker image and container name. Some of the Docker commands below use image name as the command line argument and some others use the container name as the command line argument. Use the context to understand whether it is a Docker image or a container.

Docker Image Layers

There are multiple ways to build a Docker image. Mostly it is built with a Dockerfile. These days if you are migrating legacy applications into containers, you will tend to use the project management tools such as Maven for Java, SBT for Scala, etc for building your Docker images as well. These project management tools come with plugins that can magically produce Docker images for you. When you do that, you don’t have complete control over the way the Docker images are built. Whether you are building Docker images using Dockerfile or through other tools, it is imperative to have a look at the Docker image layers and make sure that nothing extra is hiding in your images. The following command can be used to find various layers of a Docker image.

$ docker history $CONTAINER_NAME
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
b3a3bceb4943        8 hours ago         /bin/sh -c #(nop)  CMD ["main.py"]              0B                  
ace29865523b        8 hours ago         /bin/sh -c #(nop)  ENTRYPOINT ["python"]        0B                  
0a4848444aa9        8 hours ago         /bin/sh -c pip install -r /app/requirements.…   10.7MB              
2a7c3ce7ca45        8 hours ago         /bin/sh -c #(nop) WORKDIR /app                  0B                  
14b5b37c636e        8 hours ago         /bin/sh -c #(nop) COPY dir:be5956516e46348e4…   194B                
a93594ce93e7        10 days ago         /bin/sh -c #(nop)  CMD ["python3"]              0B                  
<missing>           10 days ago         /bin/sh -c set -ex;   wget -O get-pip.py 'ht…   6.04MB              
<missing>           10 days ago         /bin/sh -c #(nop)  ENV PYTHON_PIP_VERSION=19…   0B                  
<missing>           10 days ago         /bin/sh -c cd /usr/local/bin  && ln -s idle3…   32B                 
<missing>           10 days ago         /bin/sh -c set -ex  && apk add --no-cache --…   78.8MB              
<missing>           10 days ago         /bin/sh -c #(nop)  ENV PYTHON_VERSION=3.7.3     0B                  
<missing>           4 weeks ago         /bin/sh -c #(nop)  ENV GPG_KEY=0D96DF4D4110E…   0B                  
<missing>           4 weeks ago         /bin/sh -c apk add --no-cache ca-certificates   551kB               
<missing>           4 weeks ago         /bin/sh -c #(nop)  ENV LANG=C.UTF-8             0B                  
<missing>           4 weeks ago         /bin/sh -c #(nop)  ENV PATH=/usr/local/bin:/…   0B                  
<missing>           4 weeks ago         /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B                  
<missing>           4 weeks ago         /bin/sh -c #(nop) ADD file:88875982b0512a9d0…   5.53MB

These individual layers give lots of information about the image as a whole and by looking at it, you will be able to combine multiple layers to save image size, find out the spurious files got into the image, etc.

Packages in the OS

Every OS will have an application/package manager that is used for installation, configuration, and removal of the applications for that particular OS. In the case of Alpine Linux, it is apk. It is important to make sure that your Docker image does not have any unwanted packages installed which are not required for the proper functioning of your application running in the Docker container. If found, it is better to remove such packages early on. The following method can be adopted to find out the list of packages present in your running Alpine Linux Docker Container.

$ docker exec -it $CONTAINER_NAME apk list
WARNING: Ignoring APKINDEX.b89edf6e.tar.gz: No such file or directory
WARNING: Ignoring APKINDEX.737f7e01.tar.gz: No such file or directory
sqlite-libs-3.26.0-r3 x86_64 {sqlite} (Public-Domain) [installed]
krb5-libs-1.15.5-r0 x86_64 {krb5} (MIT) [installed]
musl-1.1.20-r4 x86_64 {musl} (MIT) [installed]
libbz2-1.0.6-r6 x86_64 {bzip2} (BSD) [installed]
libcom_err-1.44.5-r0 x86_64 {e2fsprogs} (GPL-2.0-or-later LGPL-2.0 BSD-3-Clause MIT) [installed]
gdbm-1.13-r1 x86_64 {gdbm} (GPL) [installed]
ncurses-libs-6.1_p20190105-r0 x86_64 {ncurses} (MIT) [installed]
zlib-1.2.11-r1 x86_64 {zlib} (zlib) [installed]
keyutils-libs-1.6-r0 x86_64 {keyutils} (GPL-2.0-or-later LGPL-2.0-or-later) [installed]
apk-tools-2.10.3-r1 x86_64 {apk-tools} (GPL2) [installed]
libintl-0.19.8.1-r4 x86_64 {gettext} (LGPL-2.1+) [installed]
readline-7.0.003-r1 x86_64 {readline} (GPL) [installed]
musl-utils-1.1.20-r4 x86_64 {musl} (MIT BSD GPL2+) [installed]
libssl1.1-1.1.1b-r1 x86_64 {openssl} (OpenSSL) [installed]
ncurses-terminfo-base-6.1_p20190105-r0 x86_64 {ncurses} (MIT) [installed]
alpine-baselayout-3.1.0-r3 x86_64 {alpine-baselayout} (GPL-2.0) [installed]
xz-libs-5.2.4-r0 x86_64 {xz} (GPL-2.0-or-later Public-Domain) [installed]
ca-certificates-20190108-r0 x86_64 {ca-certificates} (MPL-2.0 GPL-2.0-or-later) [installed]
libverto-0.3.0-r1 x86_64 {libverto} (MIT) [installed]
alpine-keys-2.1-r1 x86_64 {alpine-keys} (MIT) [installed]
libnsl-1.2.0-r0 x86_64 {libnsl} (LGPL-2.0-or-later) [installed]
busybox-1.29.3-r10 x86_64 {busybox} (GPL-2.0) [installed]
libuuid-2.33-r0 x86_64 {util-linux} (GPL-2.0 GPL-2.0-or-later LGPL-2.0-or-later BSD Public-Domain) [installed]
libtirpc-1.0.3-r0 x86_64 {libtirpc} (BSD-3-Clause) [installed]
scanelf-1.2.3-r0 x86_64 {pax-utils} (GPL-2.0) [installed]
.python-rundeps-0 noarch {.python-rundeps} () [installed]
libc-utils-0.7.1-r0 x86_64 {libc-dev} (BSD) [installed]
libffi-3.2.1-r6 x86_64 {libffi} (MIT) [installed]
libtls-standalone-2.7.4-r6 x86_64 {libtls-standalone} (ISC) [installed]
ssl_client-1.29.3-r10 x86_64 {busybox} (GPL-2.0) [installed]
krb5-conf-1.0-r1 x86_64 {krb5-conf} (MIT) [installed]
ncurses-terminfo-6.1_p20190105-r0 x86_64 {ncurses} (MIT) [installed]
expat-2.2.6-r0 x86_64 {expat} (MIT) [installed]
ca-certificates-cacert-20190108-r0 x86_64 {ca-certificates} (MPL-2.0 GPL-2.0-or-later) [installed]
libcrypto1.1-1.1.1b-r1 x86_64 {openssl} (OpenSSL) [installed]

Monitoring the list of packages on a regular basis through the continuous integration (CI) and continuous delivery (CD) tools is the best way to see whether any unwanted packages are creeping into your Docker images.

Shell Access

The easiest way to get access to a Docker container is through its native shell. Generally, Linux users use bash shell but in Alpine Linux, you need to install it explicitly. But there is another option as it comes by default which is the ash shell. You can get into an Alpine Linux based Docker container using the following method.

$ docker exec -it $CONTAINER_NAME /bin/ash
/app # ls -al
total 16
drwxr-xr-x    2 root     root          4096 Apr  7 10:19 .
drwxr-xr-x    1 root     root          4096 Apr  7 13:49 ..
-rw-r--r--    1 root     root           180 Apr  7 10:19 main.py
-rw-r--r--    1 root     root            14 Apr  7 08:51 requirements.txt
/app # cat main.py 
from flask import Flask
app = Flask(__name__)@app.route('/')
def hello_world():
    return 'Hello from Flask'if __name__ == '__main__':
    app.run(debug=False,host='0.0.0.0')
/app # cat requirements.txt 
Flask==0.10.1

Typically these shells are capable of running commands to list files, change the mode of the files and with these two, you will be able to execute your script files. It is a best practice to disable the shell if that is a possibility.

Environment Variables

In many organizations, applications are built by strictly following 12 Factor Application methodologies. The principles behind a 12-factor application are really good but many times you will get caught by the store config in the environment principle. If you store your database connection strings, security tokens, user names, and passwords, in the environment, you are inviting adversities. You can take a look at the environment variables defined in your running Docker container using the following command.

$ docker exec -it $CONTAINER_NAME env
PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=00a444a23ead
TERM=xterm
LANG=C.UTF-8
GPG_KEY=0D96DF4D4110E5C43FBFB17F2D347EA6AA65421D
PYTHON_VERSION=3.7.3
PYTHON_PIP_VERSION=19.0.3
HOME=/root

It is extremely critical that you are not storing any of your application or system secrets in clear text in any of these environment variables.

File System Contents

Many times people act clever and embed user names and passwords in binary files and package it along with the code. You are going to get a shock of your life when you know that anybody who can run some export commands can take away your files in your Docker container. Using the following command, you can export the files in your Docker container as a tar file.

$ docker export $CONTAINER_NAME > ${CONTAINER_NAME}_files.tar
$ ls -al
total 230152
drwxr-xr-x  5 RajT  staff        160 Apr  7 16:00 .
drwxr-xr-x  8 RajT  staff        256 Apr  7 09:52 ..
-rw-r--r--  1 RajT  staff        136 Apr  7 11:19 Dockerfile
drwxr-xr-x  4 RajT  staff        128 Apr  7 10:44 app
-rw-r--r--  1 RajT  staff  102371328 Apr  7 16:00 flask_files.tar

If you have any ability to obfuscate your source code and bake that into your Docker images, that is a better option. Applications built with purely interpreted languages such as Python have issues here as you have to depend on third-party tools to accomplish that feat. Compiled distribution files such as the ones built using Go, C, C++, etc are free from this particular issue to a great extent.

Applications write critical information into log files. Personally Identifiable Information (PII), and other secrets MUST not be written to log files in clear text. It is a good practice to take a random sampling of the log files and do spot checks to make sure that none of them are having anything suspicious. Log analysis tools come handy to take care of these kinds of issues.

Running Processes

In an environment where applications are running, the processes give out a lot of information that hackers love to exploit. Docker also provides a way to do the same. The /proc directory contents of a Docker container gives the details of its running processes. The following command can be used to get the list of files in the /proc directory of a running Docker container.

$ docker exec -it $CONTAINER_NAME ls -al /proc
total 4
dr-xr-xr-x  240 root     root             0 Apr  7 14:36 .
drwxr-xr-x    1 root     root          4096 Apr  7 14:36 ..
dr-xr-xr-x    9 root     root             0 Apr  7 14:36 1
dr-xr-xr-x    9 root     root             0 Apr  7 15:09 13
drwxrwxrwt    2 root     root            40 Apr  7 14:36 acpi
-r--r--r--    1 root     root             0 Apr  7 15:09 buddyinfo
dr-xr-xr-x    4 root     root             0 Apr  7 14:36 bus
-r--r--r--    1 root     root             0 Apr  7 15:09 cgroups
-r--r--r--    1 root     root             0 Apr  7 15:09 cmdline
-r--r--r--    1 root     root         23709 Apr  7 15:09 config.gz
-r--r--r--    1 root     root             0 Apr  7 15:09 consoles
-r--r--r--    1 root     root             0 Apr  7 15:09 cpuinfo
-r--r--r--    1 root     root             0 Apr  7 15:09 crypto
-r--r--r--    1 root     root             0 Apr  7 15:09 devices
-r--r--r--    1 root     root             0 Apr  7 15:09 diskstats
-r--r--r--    1 root     root             0 Apr  7 15:09 dma
dr-xr-xr-x    2 root     root             0 Apr  7 15:09 driver
-r--r--r--    1 root     root             0 Apr  7 15:09 execdomains
-r--r--r--    1 root     root             0 Apr  7 15:09 fb
-r--r--r--    1 root     root             0 Apr  7 15:09 filesystems
dr-xr-xr-x    8 root     root             0 Apr  7 14:36 fs
-r--r--r--    1 root     root             0 Apr  7 15:09 interrupts
-r--r--r--    1 root     root             0 Apr  7 15:09 iomem
-r--r--r--    1 root     root             0 Apr  7 15:09 ioports
dr-xr-xr-x   31 root     root             0 Apr  7 14:36 irq
-r--r--r--    1 root     root             0 Apr  7 15:09 kallsyms
crw-rw-rw-    1 root     root        1,   3 Apr  7 14:36 kcore
-r--r--r--    1 root     root             0 Apr  7 15:09 key-users
crw-rw-rw-    1 root     root        1,   3 Apr  7 14:36 keys
-r--------    1 root     root             0 Apr  7 15:09 kmsg
-r--------    1 root     root             0 Apr  7 15:09 kpagecgroup
-r--------    1 root     root             0 Apr  7 15:09 kpagecount
-r--------    1 root     root             0 Apr  7 15:09 kpageflags
-r--r--r--    1 root     root             0 Apr  7 15:09 loadavg
-r--r--r--    1 root     root             0 Apr  7 15:09 locks
-r--r--r--    1 root     root             0 Apr  7 15:09 meminfo
-r--r--r--    1 root     root             0 Apr  7 15:09 misc
-r--r--r--    1 root     root             0 Apr  7 15:09 modules
lrwxrwxrwx    1 root     root            11 Apr  7 15:09 mounts -> self/mounts
dr-xr-xr-x    2 root     root             0 Apr  7 15:09 mpt
-rw-r--r--    1 root     root             0 Apr  7 15:09 mtrr
lrwxrwxrwx    1 root     root             8 Apr  7 15:09 net -> self/net
-r--r--r--    1 root     root             0 Apr  7 15:09 pagetypeinfo
-r--r--r--    1 root     root             0 Apr  7 15:09 partitions
crw-rw-rw-    1 root     root        1,   3 Apr  7 14:36 sched_debug
lrwxrwxrwx    1 root     root             0 Apr  7 14:36 self -> 13
-rw-------    1 root     root             0 Apr  7 15:09 slabinfo
-r--r--r--    1 root     root             0 Apr  7 15:09 softirqs
-r--r--r--    1 root     root             0 Apr  7 15:09 stat
-r--r--r--    1 root     root             0 Apr  7 15:09 swaps
dr-xr-xr-x    1 root     root             0 Apr  7 14:36 sys
--w-------    1 root     root             0 Apr  7 14:36 sysrq-trigger
dr-xr-xr-x    2 root     root             0 Apr  7 15:09 sysvipc
lrwxrwxrwx    1 root     root             0 Apr  7 14:36 thread-self -> 13/task/13
crw-rw-rw-    1 root     root        1,   3 Apr  7 14:36 timer_list
dr-xr-xr-x    4 root     root             0 Apr  7 15:09 tty
-r--r--r--    1 root     root             0 Apr  7 15:09 uptime
-r--r--r--    1 root     root             0 Apr  7 15:09 version
-r--------    1 root     root             0 Apr  7 15:09 vmallocinfo
-r--r--r--    1 root     root             0 Apr  7 15:09 vmstat
-r--r--r--    1 root     root             0 Apr  7 15:09 zoneinfo

Ordinary users may not be able to get much out of this information but there are really sophisticated adversaries looking for opportunities to break in. It is obvious to make sure that these contents are to be audited by the internal security professionals before the information security certification process.

Docker Objects

In addition to the OS level information that can be retrieved using very obvious methods such as OS commands and utilities, Docker also provides utilities such as docker inspect to inspect the Docker container’s inner state. It returns a JSON object as the response and using JSON parsing utilities you can extract the precise information that you want.

$ # The following command gives out the complete output of all the objects
$ docker inspect $CONTAINER_NAME 
$ # The following command gives out the configuration object
$ docker inspect --format='{{json .Config}}' $CONTAINER_NAME
{
  "Hostname": "00a444a23ead",
  "Domainname": "",
  "User": "",
  "AttachStdin": false,
  "AttachStdout": false,
  "AttachStderr": false,
  "ExposedPorts": {
    "5000/tcp": {}
  },
  "Tty": false,
  "OpenStdin": false,
  "StdinOnce": false,
  "Env": [
    "PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
    "LANG=C.UTF-8",
    "GPG_KEY=0D96DF4D4110E5C43FBFB17F2D347EA6AA65421D",
    "PYTHON_VERSION=3.7.3",
    "PYTHON_PIP_VERSION=19.0.3"
  ],
  "Cmd": [
    "main.py"
  ],
  "ArgsEscaped": true,
  "Image": "flask",
  "Volumes": null,
  "WorkingDir": "/app",
  "Entrypoint": [
    "python"
  ],
  "OnBuild": null,
  "Labels": {}
}

A complete inspection of the output produced by the docker inspect command and making sure that there are no important secrets hiding in there is ideal.

Docker CLI to K8S

It is very common to orchestrate Docker containers using K8S running in GCP. It is not very easy to get access to the running containers but people and systems with the right or wrong level of access can get all the information you want from a Docker container. You can use the Docker CLI to K8S — kubectl — which is the command line tool that can also be used to interact with containers orchestrated using K8S. Anyone who is familiar with K8S and Docker will be able to exploit the Docker container vulnerabilities even when it is running in K8S.

Conclusion

It is important to know what goes into your Docker image. Slim Docker images are ideal that packages your application code to have better performing applications. To make it slim, it is imperative to include only the required components in a Docker image defined by the Dockerfile. Even if you take these precautions when preparing the Docker images, when they are running, the Docker containers can expose lots of crucial information that will make the application vulnerable. This story covered some of the obvious places to look for finding such vulnerabilities. The solutions to some of the critical elements discussed are obvious but you never know what are the other dangers lurking in blind spots. To prevent any kind of critical information leak, a constant vigil is required starting from the early days of SDLC until the application is decommissioned. The buck stops in your container.

Share:FacebookX
Join the discussion
Tech Chronicles
RSS
Follow by Email
LinkedIn
Share