Thoughts on Attending and Presenting at Linux Security Summit North America 2019

November 7, 2019 | 6 minute read
Text Size 100%:

Oracle Linux kernel developer Tom Hromatka attended Linux Security Summit NA 2019. In this blog post, Tom discusses the presentation that he gave as well as other talks he found interesting.

Linux Security Summit North America 2019

I was one of the lucky attendees at the Linux Security Summit North America 2019 conference in sunny San Diego from August 19th through the 21st.

Three major topics dominated this year's agenda - trusted computing, containers, and overall kernel security. This was largely my first interaction with trusted computing and hardware attestation, so it was very interesting to hear about all of the innovative work going on in this area.

My Presentation - The Why and How of Libseccomp

For 2019, LSS added three tutorial sessions to the schedule. These 90-minute talks were envisioned to be interactive and provide more in-depth details of a given technology.

Paul Moore (co-maintainer of libseccomp) and I presented the first tutorial of the conference. We dedicated the first 20 minutes or so to a slide show introduction to the technology. Paul has given various flavors of this talk before, and he delivered the "why" part of the talk with a brief history of seccomp and libseccomp. He has a charismatic and entertaining delivery that can captivate an audience on even the driest of subjects - which seccomp is not :). I took over with the "how" portion of the discussion and jumped right in with a comparison of white- vs blacklists. (Spoiler - if security is of the utmost concern, I recommend a whitelist.) I briefly touched on other seccomp considerations such as supporting additional architectures (x86_64, x32, etc.), strings in seccomp, and parameter filtering pitfalls.

The bulk of our timeslot was then spent writing a seccomp/libseccomp filter by hand. My goal was to highlight how easy it is to write a filter while simultaneously demonstrating some of the pitfalls (e.g. string handling) and how to debug them. In hindsight, this was a slightly crazy idea as many, many things could have gone horribly wrong.

I had a rough plan of the program we were going to write and had tested it out beforehand. But like all good plans, no battle plan survives first contact with the enemy. Here is what we ended up writing. My laptop behaved differently at the conference than it did at home which led to more involved debugging than I had envisioned. I admit that I did want some live debugging, but... not that much. (I think the cause of the behavior differences was because I had done my testing at home using STDERR, but I inadvertently switched to using STDOUT at the conference.) Ultimately though these issues were the exact catalyst I was looking for, and audience participation soared. By the end of the talk I had the attention of the entire room and many, many people were actively throwing out ideas. There was no shortage of great ideas on how to fix the problem and perhaps more importantly, how to debug the problem.

Afterward, a large number of people came up and thanked us for a fun talk. Several said that they were running the test program on their laptops while I was writing it and trying to actively debug it themselves. All in all, the talk didn't go exactly as I had envisioned, but perhaps that is for the better. The audience was amazing, and I sure had a lot of fun.

Making Containers Safer - Stéphane Graber and Christian Brauner

Stéphane and Christian are two of the lead engineers working on LXC for Canonical. They are both intelligent and often working on the forefront of containers, so when they make an upstream proposal, it's wise to pay attention.

In this talk, they mentioned several things they have worked on lately to improve kernel and container security:

  • Their users tend to run a single app in a container, rather than an entire distro. Thus, they can really lock down security via seccomp, SELinux, and cgroups to grant the bare minimum of permissions to run the application
  • LXC supports unprivileged containers launched by an unprivileged user, but there are still some limiitations. Stay tuned for patches and improvements from them :)
    • Multiple users within such a container is difficult
    • Several helper binaries are required
  • Christian again reemphasized the importance of using unprivileged containers when possible and listed several CVEs that would have been ineffective against an unprivileged container
  • They spent quite a bit of time discussing the newly-added seccomp/libseccomp user notification feature. (I worked on the libseccomp side of it.)
    • They are considering adding a "keep running the kernel filter" option to the user notification filter

Keynote: Retrospective: 26 Years of Flexible MAC - Stephen Smalley

Stephen Smalley was one of the early innovators in the Mandatory Access Control (MAC) arena (think SELinux and similar) and continues to innovate and advocate for better MAC solutions.

Stephen presented a amazingly detailed and lengthy history from ~1999 through today on the history of MACs in computing. He touched on early NSA work with closed source OSes and the NSA's inability to gain traction there. These failures drove the NSA to look at open source OSes, and early experiments with the University of Utah and the OS they maintained proved the viability of a MAC. SELinux work started shortly after that and was added to Linux in 2003.

Stephen applauded the android work as a good example of how to apply a MAC. Android is 100% confined+enforcing and has a large automated validation and testing suite.

Going forward, Stephen said that MACs are being effectively used by higher-level services and emerging technologies. For better security, this is critical.

Application Whitelisting - Steven Grubb

Steve Grubb is working on a rather novel approach to improve security. He's working on a daemon, fapolicyd, that can whitelist files on the system.

His introduction quickly spelled out the problem space. Antivirus is an effective blacklisting approach. It can identify untrusted files and rapidly neutralize them. fapolicyd is effectively the opposite. A sysadmin should generally know the expected files that will be on the system and can create an application whitelist based upon these known files.

He then went on a small tangent showing how easy it is to hijack a Python process and start up a webserver without touching the disk.

fapolicyd uses seccomp to restrict execve access. Another quick demo showed how fapolicyd will allow /bin/ls to run, but a copy of it in /tmp was blocked.

It's an interesting project in its early stages, and I'm eager to see how it progresses, so I started following it on github.

How to Write a Linux Security Module - Casey Schaufler

Casey gave the third and final tutorial of the conference on how (and why) to write a Linux Security Module (LSM). As an aside, I had lunch with Casey prior to his presentation, and he good-naturedly said that he wasn't "crazy enough" to write software live in front of a large audience. Hmmm :). Anyway...

My key takeaways from this tutorial: * Why write your own LSM? You may have unique things you want to check beyond what SELinux or Apparmor are checking. Perhaps there's one little thing that your LSM can do... and do well * One LSM cannot override another LSM's denial. In fact, once a check fails, no other LSMs are run * If the check can be readily done in userspace, do it there. This includes LSM logic * You only need to implement the LSM hooks you are interested in

Kernel Self-Protection Project - Kees Cook

Kees (kernel seccomp maintainer amongst many other things) gave another excellent talk on the status of security in the Linux kernel. His talks are usually so engaging that I don't take notes, and this one was no exception. He outlined the many security (and otherwise) fixes that have gone into the kernel over the last year. He also opined that he would love to see the kernel move away from C and replace it with rust, but he acknowledges there are a lot of challenges (both technical and human) before that could happen.

Hallway Track

As with any major Linux conference, the hallway track is every bit as invaluable as the official presentations. This was the first time I met my co-maintainer of libseccomp (Paul Moore) in person, and we were able to meet up a few times to talk seccomp/libseccomp and their roadmap going forward.

I was lucky to be able to spend some time with several of the presenters, talking containers, seccomp, cgroups and whatever other topics we had in common.

And of course I talked seccomp with many conference attendees and gladly offered my assistance in getting their seccomp filters up and running.


This was my first LSS and hopefully not my last. I really enjoyed my time with the outstanding conference attendees, and the conversations (both formal and informal) were excellent. In summary, I learned a ton, ate way too much really good food, and met many intelligent and wonderful people. I hope to see you at LSS 2020!

Tom Hromatka

Previous Post

Join Wim Coekaerts, Oracle SVP, for a webinar: What you need to know about Oracle Autonomous Linux

Guest Author | 1 min read

Next Post

Unified Management for Oracle Linux Cloud Native Environment

Avi Miller | 4 min read