DTrace a Docker Container

January 16, 2019 | 4 minute read
Text Size 100%:

Content for this blog entry was provided by Tomas Jedlicka and Eugene Loh

Software containers have standardized how software is shipped, making it simpler to run an application with its expected dependencies on a wide variety of platforms.

Here, we illustrate use of DTrace on a host system to observe activity within a Docker container, running on Oracle Linux using runC.  The central idea is that DTrace predicates can limit tracing on the host system to children of the Docker container process.

Note that a runC-like container runtime shares a single kernel instance across all containers.  Some other container implementations would not work with this same DTrace script.  An example is Clear Containers, which creates lightweight virtual machines with their own kernel instances.

System Setup

1.  Install DTrace:

    # yum install dtrace-utils

2. Install and start Docker:

Make sure that the ol7_addons repo is enabled, for example by making sure in /etc/yum.repos.d/public-yum-ol7.repo that [ol7_addons] has enabled=1.  Then:

    # yum install docker-engine
    # systemctl start docker
    # systemctl enable docker
    Created symlink [...]
    # systemctl status docker
      docker.service - Docker Application Container Engine
    Hint: Some lines were ellipsized, use -l to show in full.

3. Get an image:

In this example, we use the Oracle Container Registry. You might first have to log into the website at https://container-registry.oracle.com to accept license agreements.

     # docker login container-registry.oracle.com
    Username: myname
    Login Succeeded
    # docker pull container-registry.oracle.com/os/oraclelinux
    Using default tag: latest


Running in a container

We run our container, using -it to allocate a pseudo-TTY, creating an interactive bash shell:

   # docker run --rm -it container-registry.oracle.com/os/oraclelinux

From the host system, we can get the container ID and thereby also the process ID (PID) of the bash shell running in the container.

    # docker ps -q
    # docker ps -q | xargs docker inspect --format ""
    # ps -p 12345 -o pid,ppid,command
    12345 12340 /bin/bash
    # ps -p 12340 -o pid,cmd
      PID  CMD
    12340  docker-containerd-shim -namespace moby -workdir /var/lib/docker/cont

The bash command has pid 12345 and its parent is docker-containerd-shim, whose pid is 12340.


Using DTrace to observe activities in the container

While we manually execute commands in the container's bash shell, we can observe those activities from the host system.

The key is that DTrace allows predicates to filter tracing, and progenyof() allows us to trace only processes that are progeny (children, grandchildren, etc.) of the pid of interest.

Consider the D script syscalls.d:

    #!/usr/sbin/dtrace -s
    syscall:::                              /* trace system calls */
    / progenyof($1) /                       /* only progeny of specified pid */
    {                                       /* count occurrences */
      @[pid, execname, probefunc] = count();

We can run this script on the host system, specifying the pid on the command line, to study system calls in the container, aggregating results that are reported when the script is terminated.

    # ./syscalls.d 12345

As with many DTrace scripts, which are short though powerful, we could just run a one-liner.

    # dtrace -n 'syscall::: /progenyof($1)/ {@[pid,execname,probefunc]=count()}' 12345

Here is the example output:

    dtrace: script './syscalls.d' matched 638 probes

        [...]         13012  ls       close              26         13012  ls       read               28         13012  ls       mprotect           36         13012  ls       newlstat           38         13012  ls       open               42         13012  bash     rt_sigaction       44         12345  bash     rt_sigaction       48         13012  ls       mmap               54         12345  bash     rt_sigprocmask     58         13012  ls       rt_sigaction       76

We can also look exclusively at read and write system calls.  Since argument 2 to these calls is the size of the I/O operation, we can form histograms of these sizes:

    #!/usr/sbin/dtrace -s
    syscall::read:entry,             /* trace read and write system calls */
    /progenyof($1) && arg2 > 1024/   /* only progeny of the specified pid */
      @[probefunc] = quantize(arg2);

This script filters not only on progeny of the specified pid, but also on large (> 1 Kbyte) transfers. While this script is running on the host, we execute the following copy in the container:

    [root@4c80fafde812 /]# dd if=/dev/zero of=/dev/null bs=1M count=10
    10+0 records in
    10+0 records out
    10485760 bytes (10 MB) copied, 0.00237154 s, 4.4 GB/s
    [root@4c80fafde812 /]#

Upon completion, we can terminate the script on the host:

    # ./large_rw.d 12345
    dtrace: script './large_rw.d' matched 2 probes

      read                                                              value  ------------- Distribution ------------- count                   524288 |                                         0                      1048576 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 10                     2097152 |                                         0        
      write                                                             value  ------------- Distribution ------------- count                   524288 |                                         0                      1048576 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 10                     2097152 |                                         0     

DTrace confirms that there were ten 1-Mbyte read and write transfers in the container.


DTrace is a simple-yet-powerful tracing tool.  Using a progenyof() predicate, it can focus exclusively on processes running within a Docker container.  Using D syntax, the tracing can be filtered, and data can be selected, aggregated, and otherwise manipulated.

Docker documentation can be found at https://docs.docker.com
The Oracle Container Registry is at https://container-registry.oracle.com
An earlier post https://blogs.oracle.com/linux/dtrace-on-linux%3a-an-update discussed the port of DTrace to Linux.

Elena Zannoni

Elena Zannoni is a Senior Director in Oracle, responsible for Linux Toolchain and tracing development.

Previous Post

BPF In Depth: Communicating with Userspace

Alan Maguire | 9 min read

Next Post

BPF In Depth: Building BPF Programs

Alan Maguire | 7 min read