Oracle Linux kernel developer Alan Maguire presents this six-part series on BPF, wherein he presents an in depth look at the kernel's "Berkeley Packet Filter" -- a useful and extensible kernel function for much more than packet filtering.
Here I'm going to describe how I set up my programming environment to build BPF programs. The advice is mostly based around using Linux UEK5, which is based on a 4.14 Linux kernel, so a bit of adjustment will be needed for other distros. Note - I'm not going to talk about BCC (the BPF Compiler Collection) here; for UEK5 that extra step involves building BCC from source. BCC isn't required to build BPF programs - clang/LLVM support a BPF target, so what we're aiming for here is to compile and use BPF programs. BCC however is a great resource for programs and provides python bindings and much more.
First, verify the kernel you are working with has the following configuration options enabled:
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_NET_CLS_BPF=m
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_JIT=y
CONFIG_HAVE_BPF_JIT=y
CONFIG_BPF_EVENTS=y
All of these are enabled for our latest release based on Linux kernel 4.14, UEK5. To check these values for your running kernel:
# grep BPF /boot/config-`uname -r`
To build BPF programs, add LLVM and clang packages. clang is used to compile C programs to BPF bytecodes, and to ensure your version supports bpf, run "llc --version"; BPF should be listed as a registered target. To support BPF compilation, clang should be > version, 3.4.0 LLVM > version 3.7.1, according to http://prototype-kernel.readthedocs.io/en/latest/networking/XDP/end-user/build_environment.html . For UEK5, you can install them from the developer EPEL yum repository. Note; to use the latest LLVM/clang, "scl enable rh-dotnet20 bash" must be run. For UEK5
# yum install -y yum-utils
# yum-config-manager --add-repo=http://yum.oracle.com/public-yum-ol7.repo
# yum-config-manager --enable ol7_developer --enable ol7_developer_EPEL
# yum install -y rh-dotnet20-clang rh-dotnet20-llvm
# scl enable rh-dotnet20 bash
# llc --version |grep bpf
If you are using tc to build classifiers/actions via BPF, or want to use BPF to manage lightweight tunnel encapsulation for routes, you will need up-to-date versions of iproute and tc; i.e. 4.14-based versions, which support interaction with BPF programs. Follow those links for the UEK5 packages; for other distros you can also build these from source if needs be.
# yum-config-manager --enable ol7_UEKR5
# yum install -y iproute iproute-tc
Finally, to build BPF programs, you will need an up-to-date kernel development package to compile against, e.g. the UEK5 kernel-uek-devel or kernel-uek-headers package, since the headers shipped with UEK5 in the kernel-headers do not contain the 4.14 definitions for the BPF syscall etc.
# yum install -y kernel-uek-devel
Warning - installing kernel-uek-headers installs updated headers in /usr/include and can cause compatibility issues, so it is often best avoided in production environments. For an alternative approach to updating the header files in /usr/include using kernel-uek-devel, read on. The kernel-uek-devel package provides kernel headers and makefiles sufficient to build modules against the kernel package, and has nearly everything we need to build BPF programs; the below shows us how to use it and add the extra pieces.
This is mostly specific to my needs, but if you want to build BPF programs outside of the kernel tree while having them compatible with samples/bpf (so hopefully you can push them upstream later!), read on...
Once the above dependencies have been installed, you are ready to start building BPF programs. However, it's important to ensure they are compiled against the right headers with all the latest BPF definitions. The approach I use is to point compilation at the kernel-uek-devel headers from /usr/src/kernels/
For BPF projects, I use a directory structure as follows:
bpf/
include/
Makefile
user/
I like to mirror the samples/bpf functionality - in particular I want to be able to use bpf.c/bpf_load.c as these simplify BPF interactions by providing code to scan a BPF program and load ELF sections as maps and programs. Loading BPF programs - when not using tc or iproute for configuring lightweight tunnels, which both have BPF integration - is a pain. Explaining each subdirectory:
This approach minimizes dependencies by providing local copies of some of the samples and tools .c and .h files ; we provide our own because kernel-uek-devel does not deliver these files. Then we point compilation at the kernel headers from /usr/src/kernels/
The "bpf" subdirectory is where BPF programs are built with LLVM/clang, and to simplify the build process I add local copies of bpf_helpers.h and bpf_endian.h to the include/ directory. Also added here is linux/types.h; a copy of tools/include/linux/types.h. Here is the full bpf/Makefile:I use; in this case we are building one object; socket_filter_kernel.o; more can be added to OBJS as needed.
# SPDX-License-Identifier: GPL-2.0
#
# Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License version 2
# as published by the Free Software Foundation.
#
# Build bpf code (kernel) out-of-tree by referencing local copies of
# bpf .h files along with headers from kernel source tree.
# Creates similar environment to that used by samples/bpf by adding
# ../include/[bpf_endian.h,bpf_helpers.h,linux/types.h]. The latter is
# used to get definitions for u64, u32 etc which are needed by other kernel
# headers.
#
# - ../include/bpf_helpers.h is a copy of tools/testing/selftest/bpf/bpf_helpers.h
# - ../include/bpf_endian.h is a copy of tools/testing/selftest/bpf/bpf_endian.h
# - ../include/linux/types.h is a copy of tools/include/linux/types.h
#
#
# Assumptions:
#
# - kernel-uek-devel package or equivalent has installed (partial) source
# tree in /usr/src/kernels/`uname -r`
#
# - llc/clang are available and support "bpf" target; check with "llc --verison"
#
OBJS = socket_filter_kernel.o
LLC ?= llc
CLANG ?= clang
INC_FLAGS = -nostdinc -isystem `$(CLANG) -print-file-name=include`
EXTRA_CFLAGS ?= -O2 -emit-llvm
# In case up-to-date headers are not installed locally in /usr/include,
# use source build.
linuxhdrs ?= /usr/src/kernels/`uname -r`
LINUXINCLUDE = -I$(linuxhdrs)/arch/x86/include/uapi \
-I$(linuxhdrs)/arch/x86/include/generated/uapi \
-I$(linuxhdrs)/include/generated/uapi \
-I$(linuxhdrs)/include/uapi \
-I$(linuxhdrs)/include
prefix ?= /usr/local
INSTALLPATH = $(prefix)/lib/bpf
install_PROGRAM = install
install_DIR = install -dv
all: $(OBJS)
.PHONY: clean
clean:
rm -f $(OBJS)
INC_FLAGS = -nostdinc -isystem `$(CLANG) -print-file-name=include`
$(OBJS): %.o:%.c
$(CLANG) $(INC_FLAGS) \
-D__KERNEL__ -D__ASM_SYSREG_H \
-Wno-unused-value -Wno-pointer-sign \
-Wno-compare-distinct-pointer-types \
-Wno-gnu-variable-sized-type-not-at-end \
-Wno-address-of-packed-member -Wno-tautological-compare \
-Wno-unknown-warning-option \
-I../include $(LINUXINCLUDE) \
$(EXTRA_CFLAGS) -c $< -o -| $(LLC) -march=bpf -filetype=obj -o $@
install: $(OBJS)
$(install_DIR) -d $(INSTALLPATH) ; \
$(install_PROGRAM) $^ -t $(INSTALLPATH)
uninstall: $(OBJS)
rm -rf $(INSTALLPATH)
In the user/ subdirectory I add local copies of bpf.[ch], bpf_load.[ch], bpf_util.h and perf-sys.h. Here is the Makefile:
# SPDX-License-Identifier: GPL-2.0
#
# Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License version 2
# as published by the Free Software Foundation.
#
# Build bpf userspace code out-of-tree by referencing local copies of
# bpf .c and .h files.
#
# - bpf.[ch] are copies of tools/lib/bpf/bpf.[ch]
# - bpf_load.[ch] are copies are samples/bpf/bpf_load.[ch], with references
# to #include the unneeded libbpf.h removed, replaced by references to bpf.h
# - bpf_util.h is a copy of tools/testing/selftests/bpf/bpf_util.h
# - perf-sys.h is a copy of tools/perf/perf-sys.h
#
COMMONOBJS = bpf.o bpf_load.o
SOCKETFILTERPROG = socket_filter_user
SOCKETFILTEROBJ = $(SOCKETFILTERPROG).o
PROGS= $(SOCKETFILTERPROG)
OBJS= $(COMMONOBJS) $(SOCKETFILTEROBJ)
linuxhdrs ?= /usr/src/kernels/`uname -r`
LINUXINCLUDE = -I$(linuxhdrs)/arch/x86/include/uapi \
-I$(linuxhdrs)/arch/x86/include/generated/uapi \
-I$(linuxhdrs)/include/generated/uapi \
-I$(linuxhdrs)/include/uapi \
-I$(linuxhdrs)/include
prefix ?= /usr/local
INSTALLPATH = $(prefix)/bin
install_PROGRAM = install
install_DIR = install -d
LDLIBS = -lelf
all: $(SOCKETFILTERPROG)
.PHONY: clean
clean:
rm -f $(OBJS) $(PROGS)
%.o: %.c
$(CC) -g -Wno-unused-variable -I../include $(LINUXINCLUDE) -c -o $@ $< $(CFLAGS)
$(PROGS): $(OBJS)
$(CC) -g -o $@ $(@).o $(COMMONOBJS) $(CFLAGS) $(LDLIBS)
install: $(PROGS)
$(install_DIR) -d $(INSTALLPATH) ; \
$(install_PROGRAM) $^ -t $(INSTALLPATH)
uninstall: $(PROGS)
cd $(INSTALLPATH); rm -f $^
Sample code and repo can be found on Oracle's github page here https://github.com/oracle/linux-blog-sample-code/tree/bpf-test/bpf-test
So we've seen how to install dependencies and how to set things up to build BPF programs. From here you might want to build some BPF programs of your own, or install BCC and play around with it. Anyway hopefully some of this has been useful.
Thanks for reading this installment of our six part series on BPF. We hope you found it educational and useful. Questions or comments? Use the comments field below!
Stay tuned for the next installment in this series, The BPF Bytecode Verifier.
Be sure to visit the previous installments of this series on BPF, here, and stay tuned for our next blog posts! 1. BPF program types 2. BPF helper functions for those programs 3. BPF userspace communication 4. BPF program build environment