Content for this blog entry was provided by Tomas Jedlicka and Eugene Loh

Software containers have standardized how software is shipped, making it simpler to run an application with its expected dependencies on a wide variety of platforms.

Here, we illustrate use of DTrace on a host system to observe activity within a Docker container, running on Oracle Linux using runC.  The central idea is that DTrace predicates can limit tracing on the host system to children of the Docker container process.

Note that a runC-like container runtime shares a single kernel instance across all containers.  Some other container implementations would not work with this same DTrace script.  An example is Clear Containers, which creates lightweight virtual machines with their own kernel instances.

System Setup

1.  Install DTrace:

    # yum install dtrace-utils

2. Install and start Docker:

Make sure that the ol7_addons repo is enabled, for example by making sure in /etc/yum.repos.d/public-yum-ol7.repo that [ol7_addons] has enabled=1.  Then:

    # yum install docker-engine
    [...]
    Complete!
    # systemctl start docker
    # systemctl enable docker
    Created symlink [...]
    # systemctl status docker
      docker.service - Docker Application Container Engine
    [...]
    Hint: Some lines were ellipsized, use -l to show in full.

3. Get an image:

In this example, we use the Oracle Container Registry. You might first have to log into the website at https://container-registry.oracle.com to accept license agreements.

     # docker login container-registry.oracle.com
    Username: myname
    Password:
    Login Succeeded
    # docker pull container-registry.oracle.com/os/oraclelinux
    Using default tag: latest
    [...]

Running in a container

We run our container, using -it to allocate a pseudo-TTY, creating an interactive bash shell:

   # docker run --rm -it container-registry.oracle.com/os/oraclelinux

From the host system, we can get the container ID and thereby also the process ID (PID) of the bash shell running in the container.

    # docker ps -q
    4c80fafde812
    # docker ps -q | xargs docker inspect --format ""
    12345
    # ps -p 12345 -o pid,ppid,command
      PID  PPID COMMAND
    12345 12340 /bin/bash
    # ps -p 12340 -o pid,cmd
      PID  CMD
    12340  docker-containerd-shim -namespace moby -workdir /var/lib/docker/cont
    #

The bash command has pid 12345 and its parent is docker-containerd-shim, whose pid is 12340.

Using DTrace to observe activities in the container

While we manually execute commands in the container's bash shell, we can observe those activities from the host system.

The key is that DTrace allows predicates to filter tracing, and progenyof() allows us to trace only processes that are progeny (children, grandchildren, etc.) of the pid of interest.

Consider the D script syscalls.d:

    #!/usr/sbin/dtrace -s
    syscall:::                              /* trace system calls */
    / progenyof($1) /                       /* only progeny of specified pid */
    {                                       /* count occurrences */
      @[pid, execname, probefunc] = count();
    }

We can run this script on the host system, specifying the pid on the command line, to study system calls in the container, aggregating results that are reported when the script is terminated.

    # ./syscalls.d 12345

As with many DTrace scripts, which are short though powerful, we could just run a one-liner.

    # dtrace -n 'syscall::: /progenyof($1)/ {@[pid,execname,probefunc]=count()}' 12345

Here is the example output:

    dtrace: script './syscalls.d' matched 638 probes
    ^C


        [...]
        13012  ls       close              26
        13012  ls       read               28
        13012  ls       mprotect           36
        13012  ls       newlstat           38
        13012  ls       open               42
        13012  bash     rt_sigaction       44
        12345  bash     rt_sigaction       48
        13012  ls       mmap               54
        12345  bash     rt_sigprocmask     58
        13012  ls       rt_sigaction       76

We can also look exclusively at read and write system calls.  Since argument 2 to these calls is the size of the I/O operation, we can form histograms of these sizes:

    #!/usr/sbin/dtrace -s
    syscall::read:entry,             /* trace read and write system calls */
    syscall::write:entry
    /progenyof($1) && arg2 > 1024/   /* only progeny of the specified pid */
    {
      @[probefunc] = quantize(arg2);
    }

This script filters not only on progeny of the specified pid, but also on large (> 1 Kbyte) transfers. While this script is running on the host, we execute the following copy in the container:

    [root@4c80fafde812 /]# dd if=/dev/zero of=/dev/null bs=1M count=10
    10+0 records in
    10+0 records out
    10485760 bytes (10 MB) copied, 0.00237154 s, 4.4 GB/s
    [root@4c80fafde812 /]#

Upon completion, we can terminate the script on the host:

    # ./large_rw.d 12345
    dtrace: script './large_rw.d' matched 2 probes
    ^C


      read                                              
               value  ------------- Distribution ------------- count    
              524288 |                                         0        
             1048576 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 10       
             2097152 |                                         0        


      write                                             
               value  ------------- Distribution ------------- count    
              524288 |                                         0        
             1048576 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 10       
             2097152 |                                         0     

DTrace confirms that there were ten 1-Mbyte read and write transfers in the container.

Conclusions

DTrace is a simple-yet-powerful tracing tool.  Using a progenyof() predicate, it can focus exclusively on processes running within a Docker container.  Using D syntax, the tracing can be filtered, and data can be selected, aggregated, and otherwise manipulated.

Docker documentation can be found at https://docs.docker.com
The Oracle Container Registry is at https://container-registry.oracle.com
An earlier post https://blogs.oracle.com/linux/dtrace-on-linux%3a-an-update discussed the port of DTrace to Linux.