Monitoring a Host from Inside a Container

The Docker philosophy of isolation can be circumvented by mounting host directories into the container (as Datadog client does, for example) or running a container in --privileged container mode (or --share-system for systemd-nspawn). This prevents pid/network/ipc/disk/uts namespacing (or just pid/uts/ipc for nspawn), allowing access to all devices and effectively launching the process as if it were on the host.

These tools are invaluable when debugging an immutable host system such as CoreOS.

But privileged mode is not necessary if you only want access to certain parts of the host machine. For example Datadog currently launches its agent ("monitoring container") with these flags (specific to its monitoring requirements):

docker run -d --name dd-agent -h `hostname` \
  -v /var/run/docker.sock:/var/run/docker.sock -v /proc/:/host/proc/:ro \
  -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -e API_KEY={your_api_key_here} \
  datadog/docker-dd-agent

(notice the volume mounts giving read-only access to the hosts proc and cgroup directories, as well as the docker socket [to monitor the daemon])

Sysdig Cloud requires privileged mode, because it has far deeper system introspection capabilities, whilst also mounting device, process, boot, modules and user directories:

docker run --name sysdig-agent --privileged --net host --pid host \
  -e ACCESS_KEY=[ACCESS_KEY] -e TAGS=[TAGS] \
  -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev \
  -v /proc:/host/proc:ro -v /boot:/host/boot:ro \
  -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro sysdig/agent

It is also possible to add and revoke individual capabilities using --cap-add and --cap-drop.

CoreOS provide a toolbox script (distinct from the new docker-toolbox) to launch this style of container for you using systemd-nspawn instead of docker - they both run containers.

systemd-nspawn has different syntax to Docker, but the effect is still the same - the host system is shared with the container (source):

sudo systemd-nspawn \
    --directory="${machinepath}" \
  --capability=all \
  --share-system \
  --bind=/:/media/root \
  --bind=/usr:/media/root/usr \
  --bind=/run:/media/root/run \
  --user="${TOOLBOX_USER}" "$@"

In summary, you can launch a container and install debugging tools that can inspect the host (and by extension, other containers) by using Docker with specific volume mounts and/or --privileged, or CoreOS's toolbox.

n.b. my personal preference for debugging containers is Sysdig: "Think about sysdig as strace + tcpdump + htop + iftop + lsof + ...awesome sauce." - which currently looks like:

docker run -i -t --name sysdig --privileged \
  -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev \
  -v /proc:/host/proc:ro -v /boot:/host/boot:ro \
  -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro sysdig/sysdig