Notes on BPF (4) - Setting up your environment to build BPF programs

Oracle Linux kernel developer Alan Maguire presents this six-part series on BPF, wherein he presents an in depth look at the kernel's "Berkeley Packet Filter" -- a useful and extensible kernel function for much more than packet filtering.

Here I'm going to describe how I set up my programming environment to build BPF programs. The advice is mostly based around using Linux UEK5, which is based on a 4.14 Linux kernel, so a bit of adjustment will be needed for other distros. Note - I'm not going to talk about BCC (the BPF Compiler Collection) here; for UEK5 that extra step involves building BCC from source. BCC isn't required to build BPF programs - clang/LLVM support a BPF target, so what we're aiming for here is to compile and use BPF programs. BCC however is a great resource for programs and provides python bindings and much more.

Install dependencies

First, verify the kernel you are working with has the following configuration options enabled:

CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_NET_CLS_BPF=m
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_JIT=y
CONFIG_HAVE_BPF_JIT=y
CONFIG_BPF_EVENTS=y

All of these are enabled for our latest release based on Linux kernel 4.14, UEK5. To check these values for your running kernel:

# grep BPF /boot/config-`uname -r`

To build BPF programs, add LLVM and clang packages. clang is used to compile C programs to BPF bytecodes, and to ensure your version supports bpf, run "llc --version"; BPF should be listed as a registered target. To support BPF compilation, clang should be > version, 3.4.0 LLVM > version 3.7.1, according to http://prototype-kernel.readthedocs.io/en/latest/networking/XDP/end-user/build_environment.html . For UEK5, you can install them from the developer EPEL yum repository. Note; to use the latest LLVM/clang, "scl enable rh-dotnet20 bash" must be run. For UEK5

# yum install -y yum-utils
# yum-config-manager --add-repo=http://yum.oracle.com/public-yum-ol7.repo
# yum-config-manager --enable ol7_developer --enable ol7_developer_EPEL
# yum install -y rh-dotnet20-clang rh-dotnet20-llvm
# scl enable rh-dotnet20 bash
# llc --version |grep bpf

If you are using tc to build classifiers/actions via BPF, or want to use BPF to manage lightweight tunnel encapsulation for routes, you will need up-to-date versions of iproute and tc; i.e. 4.14-based versions, which support interaction with BPF programs. Follow those links for the UEK5 packages; for other distros you can also build these from source if needs be.

# yum-config-manager --enable ol7_UEKR5
# yum install -y iproute iproute-tc

Finally, to build BPF programs, you will need an up-to-date kernel development package to compile against, e.g. the UEK5 kernel-uek-devel or kernel-uek-headers package, since the headers shipped with UEK5 in the kernel-headers do not contain the 4.14 definitions for the BPF syscall etc.

# yum install -y kernel-uek-devel

Warning - installing kernel-uek-headers installs updated headers in /usr/include and can cause compatibility issues, so it is often best avoided in production environments. For an alternative approach to updating the header files in /usr/include using kernel-uek-devel, read on. The kernel-uek-devel package provides kernel headers and makefiles sufficient to build modules against the kernel package, and has nearly everything we need to build BPF programs; the below shows us how to use it and add the extra pieces.

Building BPF programs outside the kernel tree

This is mostly specific to my needs, but if you want to build BPF programs outside of the kernel tree while having them compatible with samples/bpf (so hopefully you can push them upstream later!), read on...

Once the above dependencies have been installed, you are ready to start building BPF programs. However, it's important to ensure they are compiled against the right headers with all the latest BPF definitions. The approach I use is to point compilation at the kernel-uek-devel headers from /usr/src/kernels/, and we can pick up the up-to-date BPF headers, even if out-of-date headers are still installed in /usr/include (which they often are for backwards compatibility).The only complication with doing this is that the kernel-uek-devel package does not include a few files that are needed for compilation to succeed on the kernel side, and on the user-space side there are some convenience functions etc implemented which we would like to use in our programs. Here I'll describe how I've tackled this; again this may not make sense for your situation, but there may be aspects of the Makefiles that you can re-use.

For BPF projects, I use a directory structure as follows:

bpf/
include/
Makefile
user/

I like to mirror the samples/bpf functionality - in particular I want to be able to use bpf.c/bpf_load.c as these simplify BPF interactions by providing code to scan a BPF program and load ELF sections as maps and programs. Loading BPF programs - when not using tc or iproute for configuring lightweight tunnels, which both have BPF integration - is a pain. Explaining each subdirectory:

• The "bpf" directory contains the code to be compiled into BPF programs
• "include" contains headers used to build - bpf_endian.h, bpf_ and linux/types.h (the latter is needed for u64, u32 definitions etc.), along with any common headers needed by the user and bpf subdirectories.
• "user" contains the BPF user-space code that load BPF program in the kernel and interacts with them, and uses copies of bpf_load.c/bpf_load.h, and bpf.c/bpf.h from the kernel tree to do this.

This approach minimizes dependencies by providing local copies of some of the samples and tools .c and .h files ; we provide our own because kernel-uek-devel does not deliver these files. Then we point compilation at the kernel headers from /usr/src/kernels/, and we can pick up the up-to-date BPF headers even if out-of-date headers are installed in /usr/include. Thus we can build BPF programs without having to install up-to-date kernel-uek-headers which, when installed, could cause breakage elsewhere.

Building BPF programs - kernel

The "bpf" subdirectory is where BPF programs are built with LLVM/clang, and to simplify the build process I add local copies of bpf_helpers.h and bpf_endian.h to the include/ directory. Also added here is linux/types.h; a copy of tools/include/linux/types.h. Here is the full bpf/Makefile:I use; in this case we are building one object; socket_filter_kernel.o; more can be added to OBJS as needed.

# SPDX-License-Identifier: GPL-2.0
#
# Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License version 2
# as published by the Free Software Foundation.
#
# Build bpf code (kernel) out-of-tree by referencing local copies of
# bpf .h files along with headers from kernel source tree.
# Creates similar environment to that used by samples/bpf by adding
# ../include/[bpf_endian.h,bpf_helpers.h,linux/types.h]. The latter is
# used to get definitions for u64, u32 etc which are needed by other kernel
# headers.
#
# - ../include/bpf_helpers.h is a copy of tools/testing/selftest/bpf/bpf_helpers.h
# - ../include/bpf_endian.h is a copy of tools/testing/selftest/bpf/bpf_endian.h
# - ../include/linux/types.h is a copy of tools/include/linux/types.h
#
#
# Assumptions:
#
# - kernel-uek-devel package or equivalent has installed (partial) source
#   tree in /usr/src/kernels/`uname -r`
#
# - llc/clang are available and support "bpf" target; check with "llc --verison"
#

OBJS = socket_filter_kernel.o

LLC ?= llc
CLANG ?= clang
INC_FLAGS = -nostdinc -isystem `$(CLANG) -print-file-name=include`
EXTRA_CFLAGS ?= -O2 -emit-llvm

# In case up-to-date headers are not installed locally in /usr/include,
# use source build.

linuxhdrs ?= /usr/src/kernels/`uname -r`

LINUXINCLUDE =  -I$(linuxhdrs)/arch/x86/include/uapi \
                -I$(linuxhdrs)/arch/x86/include/generated/uapi \
                -I$(linuxhdrs)/include/generated/uapi \
                -I$(linuxhdrs)/include/uapi \
                -I$(linuxhdrs)/include

prefix ?= /usr/local

INSTALLPATH = $(prefix)/lib/bpf

install_PROGRAM = install
install_DIR = install -dv

all: $(OBJS)

.PHONY: clean

clean:
        rm -f $(OBJS)

INC_FLAGS = -nostdinc -isystem `$(CLANG) -print-file-name=include`

$(OBJS):  %.o:%.c
        $(CLANG) $(INC_FLAGS) \
                -D__KERNEL__ -D__ASM_SYSREG_H \
                -Wno-unused-value -Wno-pointer-sign \
                -Wno-compare-distinct-pointer-types \
                -Wno-gnu-variable-sized-type-not-at-end \
                -Wno-address-of-packed-member -Wno-tautological-compare \
                -Wno-unknown-warning-option \
                -I../include $(LINUXINCLUDE) \
                $(EXTRA_CFLAGS) -c $< -o -| $(LLC) -march=bpf -filetype=obj -o $@

install: $(OBJS)
        $(install_DIR) -d $(INSTALLPATH) ; \
        $(install_PROGRAM) $^ -t $(INSTALLPATH)

uninstall: $(OBJS)
        rm -rf $(INSTALLPATH)

Building BPF programs - user-space

In the user/ subdirectory I add local copies of bpf.[ch], bpf_load.[ch], bpf_util.h and perf-sys.h. Here is the Makefile:

# SPDX-License-Identifier: GPL-2.0
#
# Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License version 2
# as published by the Free Software Foundation.
#
# Build bpf userspace code out-of-tree by referencing local copies of
# bpf .c and .h files.
#
# - bpf.[ch] are copies of tools/lib/bpf/bpf.[ch]
# - bpf_load.[ch] are copies are samples/bpf/bpf_load.[ch], with references
#   to #include the unneeded libbpf.h removed, replaced by references to bpf.h
# - bpf_util.h is a copy of tools/testing/selftests/bpf/bpf_util.h
# - perf-sys.h is a copy of tools/perf/perf-sys.h
#

COMMONOBJS = bpf.o bpf_load.o

SOCKETFILTERPROG = socket_filter_user
SOCKETFILTEROBJ = $(SOCKETFILTERPROG).o
PROGS= $(SOCKETFILTERPROG)
OBJS= $(COMMONOBJS) $(SOCKETFILTEROBJ)

linuxhdrs ?= /usr/src/kernels/`uname -r`

LINUXINCLUDE =  -I$(linuxhdrs)/arch/x86/include/uapi \
                -I$(linuxhdrs)/arch/x86/include/generated/uapi \
                -I$(linuxhdrs)/include/generated/uapi \
                -I$(linuxhdrs)/include/uapi \
                -I$(linuxhdrs)/include

prefix ?= /usr/local

INSTALLPATH = $(prefix)/bin

install_PROGRAM = install
install_DIR = install -d

LDLIBS = -lelf

all: $(SOCKETFILTERPROG)

.PHONY: clean

clean:
        rm -f $(OBJS) $(PROGS)

%.o: %.c
        $(CC) -g -Wno-unused-variable -I../include $(LINUXINCLUDE) -c -o $@ $< $(CFLAGS)

$(PROGS): $(OBJS)
        $(CC) -g -o $@ $(@).o $(COMMONOBJS) $(CFLAGS) $(LDLIBS)

install: $(PROGS)
        $(install_DIR) -d $(INSTALLPATH) ; \
        $(install_PROGRAM) $^ -t $(INSTALLPATH)

uninstall: $(PROGS)
        cd $(INSTALLPATH); rm -f $^

Example

Sample code and repo can be found on Oracle's github page here https://github.com/oracle/linux-blog-sample-code/tree/bpf-test/bpf-test

Summary

So we've seen how to install dependencies and how to set things up to build BPF programs. From here you might want to build some BPF programs of your own, or install BCC and play around with it. Anyway hopefully some of this has been useful.

Learning more about BPF

Thanks for reading this installment of our six part series on BPF. We hope you found it educational and useful. Questions or comments? Use the comments field below!

Stay tuned for the next installment in this series, The BPF Bytecode Verifier.

Be sure to visit the previous installments of this series on BPF, here, and stay tuned for our next blog posts! 1. BPF program types 2. BPF helper functions for those programs 3. BPF userspace communication 4. BPF program build environment