why unix | RBL service | netrs | please | ripcalc | linescroll
hosted services

hosted services

Docker has created a buzz over the last few years. More recently it has come to light and create a place for itself through inexpensive cloud provided VMs.

The notes here are for my future self as much as others as I'm sure over time my own personal infrastructure will need tweaking and I will have forgotten what I had setup initially. Much is the way with unix, things work for years before they need maintenance. Windows infrastructure is very much more needy and requires continual human poking to keep it running.

getting started

Before doing anything else, take a look at how docker is currently configured.

# docker info

Yes, that's right, everything you do should be as root. You'll not have much freedom to configure networking or start containers otherwise.

You'll get back some input such as the below, which was taken from a test VM:

Containers: 1
Images: 11
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 13
 Dirperm1 Supported: true
Execution Driver: native-0.2
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
CPUs: 1
Total Memory: 1000 MiB
Name: dockertest

The root for the containers is in /var/lib/docker/aufs, which is where docker will store all the images and instances. Expect this to grow massively.

If you don't already have docker installed, the package you shall need to apt-get install on a Debian system is:

Package: docker.io
Version: 1.6.2~dfsg1-1~bpo8+1
Installed-Size: 21493
Maintainer: Paul Tagliamonte <paultag@debian.org>
Architecture: amd64
Replaces: docker (<< 1.5~)
Depends: adduser, iptables, init-system-helpers (>= 1.18~), perl, libapparmor1 (
>= 2.6~devel), libc6 (>= 2.14), libdevmapper1.02.1 (>= 2:1.02.90), libsqlite3-0
(>= 3.5.9)
Recommends: aufs-tools, ca-certificates, cgroupfs-mount | cgroup-lite, git, xz-u
tils

images

Docker has two main concepts, the image and the container. It is relatively easy to create image templates, and once you have the image defined you can crate container instances from it, diff those to the original image as required and start/stop/destroy as needed.

This is where docker comes into its own as this is essentially what application developers require and how you can build your own PaaS infrastructure with ease.

containers

The container is the unit of grunt for your system. This is the 'jail' or the 'chroot' that your application can run within, in isolation.
You'll see the processes in your 'ps' table:

# docker top be18a828ef75
UID     PID     PPID  C  STIME   TTY   TIME       CMD
root    1052    494   0  22:16   ?     00:00:00   /usr/sbin/httpd -D FOREGROUND
...
# pstree -capl 1052
httpd,1052 -D FOREGROUND
  ├─httpd,1076 -D FOREGROUND
  ├─httpd,1077 -D FOREGROUND
  ├─httpd,1078 -D FOREGROUND
  ├─httpd,1079 -D FOREGROUND
  ├─httpd,1080 -D FOREGROUND
  ├─httpd,1081 -D FOREGROUND
  ├─httpd,1082 -D FOREGROUND
  └─httpd,1083 -D FOREGROUND

Containers are built from images using the create flag, detailed below.

dockerfile

The Dockerfile is the guts of your image configuration. This is what holds the source and the setup of what will be the container. Here's an example for a web server. I use the web server as it's a simple example of something that is easily clustered. The three least complex tasks are DNS, mail and web server. 'httpd' being a mostly readonly service is simple to setup and try.

FROM oraclelinux:6.8
MAINTAINER "Me"
RUN yum -y install httpd
COPY Dockerfile /var/www/html/index.html
EXPOSE 80
ENTRYPOINT /usr/sbin/httpd -D FOREGROUND

This example will setup a RHEL-like web server, that, in my test, shared a Debian kernel. The content in the web root will be this Dockerfile itself, which is placed there through the 'COPY' command. This is where the application setup and customisation is created. Once that is defined, the only remaining thing to do is execute the application via the 'ENTRYPOINT' command.

We can build this using the following command:

# docker build --tag="me/httpd:v1" /root/c6httpd

To test it, we can spin up a container from this image using the following:

# docker run -d --name guest -p 8080:80 me/httpd:v1

Now point your browser at port 8080 on the host and you should get the Dockerfile sent back to you as if it were a HTML (which it isn't).

If you're not happy with the container, simply remove it using the following:

# docker rm ...

Where the ellipsis is the container ID, which can be found in the output of docker ps. If the docker container has already stopped, and you want to remove it, you can do so by running:

# docker ps --filter "status=exited"

or to show all by using

# docker ps -a

The create flag can be used now we have an image: # docker create -it --name web1 -p 8080:80 me/httpd:v1

Which creates the container that we can now start:

# docker start web1

This is all very simple, efficient and secure. We don't have fully VM hardware emulation that you would have with kvm/xen/virtualbox/vmware, but there is a good level of security. There's a level of isolation that you would not otherwise have if the application was controlled with only user-level separation. Another good feature is that

# docker stats web1

Will present the network/memory/cpu load of the named containers, something that is very helpful when the host is under heavy load.

persistence

One thing you may notice from docker is that when then host machine reboots, your containers will not automatically start. To cater for this there are two methods, you can either create a system service that handles the startup of the machine, similar to daemontools, systemd unit, or build container images and run them with the --restart always argument (my preference).

In the above example we built the container from a Dockerfile, gave it a tag, then ran it. Here we run the container with the new argument to instruct its recreation upon exit.

# docker build --tag="me/httpd:v1" /root/c6httpd
# docker run -d --name guest -p 8080:80 --restart always me/httpd:v1

inter-container IP networking

The holy grail of system admin would be to have every executing part of a HTTP server run within its own container. Doing this builds upon Unix process isolation and contains an application. From my perspective it would be best to put a Tomcat application in it's own container, so each application has its own JVM, each significant FCGI has its own container too.

Doing this allows better security and you can easily spot heavy load through container spikes.

I've found the easiest way to run a HTTP setup similar to this is to make your network service HTTP a big proxy.

  +--[server1]----------------------------------------+
  |                                                   |
  |  +-------------+ +-------------+ +-------------+  |
  |  | container 2 | | container 3 | | container 4 |  |
  |  |             | |             | |             |  |
  |  | port 8002   | | port 8003   | | port 8004   |  |
  |  | /balance    | | /sales      | | /orders     |  |
  |  |             | |             | |             |  |
  |  +------+------+ +------+------+ +------+------+  |
  |         \               |               |         |
  |          ----------\    |    /---------/          |
  |                  +--+---+---+--+                  |
  |                  | container 1 |                  |
  |                  | port 80/443 |                  |
  |                  | /           |                  |
  |                  +------+------+                  |
  |                         |                         |
  |                         |                         |
  +-------------------------|-------------------------+

In Apache HTTPd I've found the easiest way to do this is as follows:

container 1

docker create --link container2:container2 --link container3:container3 --link container4:container4 --name container1 -p 80:80 -p 443:443 baseimage

The Apache HTTPd configuration file for this would be something like so:

<VirtualHost *:80>
    <Location /balance>
        ProxyPass http://${CONTAINER2_PORT_80_TCP_ADDR}:${CONTAINER2_PORT_80_TCP_PORT}/
        </Location>
    <Location /sales>
        ProxyPass http://${CONTAINER3_PORT_80_TCP_ADDR}:${CONTAINER3_PORT_80_TCP_PORT}/
    </Location>
    <Location /orders>
        ProxyPass http://${CONTAINER4_PORT_80_TCP_ADDR}:${CONTAINER4_PORT_80_TCP_PORT}/
        </Location>
</VirtualHost>

This method is labelled as deprecated. The current advise is to do the following:

docker network create string

Then supply --network string to each container you create/run, the container will be able to resolve other containers during gethostbyname. This avoids potential ephemeral port problems with environments. However, in the above example you will introduce a DNS lookup in a balancer.

inspection

You can 'inspect' a docker container configuration using docker inspect ..., but if you want to look around a running container then you can easily run

docker exec -it guest /bin/bash

This will give you a shell (providing /bin/bash exists in your image) where you can view the contents of the container.

If you want to run your container with a different entry point, then you can do so as follows:

docker run -it --name newcontainer --entrypoint=/bin/sh oldcontainer

This creates the newcontainer. If you want to add parameters, give those after oldcontainer.

mounts

One of the features of docker that I appreciate are the FS mounts that you can arbitrarily use, if you wish to make /var/www/html consistent through your containers, you can do so with:

docker run -d --name guest -p 8002:80 -v /data/content/www:/var/www/html --restart always me/httpd:v1

This will make /data/content/www on the host appear at /var/www/html in the guest container. This, I find is a very convenient way of making areas such as /var/log common for your containers and simple for fail2ban and other such log processors.

network isolation

If you wish to run a firewall within the container, the key is to start the container with the tun network device and --cap-add=NET_ADMIN and --cap-add=NET_RAW. Here's how I did it.

docker create --network network1 --device /dev/net/tun
--cap-add=NET_ADMIN --cap-add=NET_RAW --name container1 myimage

When you start the container, you can apply your network filter rules.

#!/bin/sh

PHY=`ip link show type veth | grep UP | sed -e 's/^\S\+: //g' -e 's/:.*//g' -e 's/@.*//g'`

iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT DROP
iptables -A OUTPUT -o tun0 -j ACCEPT
iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A OUTPUT -o $PHY -p udp --dport 1198 -j ACCEPT
iptables -A OUTPUT -o $PHY -p tcp --dport 1198 -j ACCEPT
iptables -A OUTPUT -o $PHY -p tcp --dport 5353 -j ACCEPT
iptables -A OUTPUT -o $PHY -p udp --dport 5353 -j ACCEPT
iptables -A OUTPUT -o $PHY -p udp --dport 53 -j ACCEPT
iptables -A OUTPUT -o $PHY -p tcp --dport 53 -j ACCEPT
iptables -A OUTPUT -o $PHY -p tcp --dport 22 -j ACCEPT

In this way we restrict all traffic other than VPN traffic tcp/1198 tcp/22 and DNS output, unless it is for the tun0 interface.

services and swarms

Initialise a swarm cluster:

how do i see how much space a docker container is using?

docker ps --size