This blog entry was contributed by Qing Zhao. Qing works on GCC in the Oracle Linux Toolchain Team, has been participating to the KSPP project for several years, and contributed several security features to GCC.

GNU logo

Introduction

We enhanced the GCC compiler to enable developers to detect more dangerous buffer overflows, by adding a new GCC option called -fstrict-flex-arrays and a corresponding warning option -Wstrict-flex-arrays.

This work was done as part of the Kernel Self Protection Project (KSPP) to which we have been participating for a few years. We’ll explain why this was necessary, how the options work, and how they help in a real case scenario, the Linux Kernel. Thanks to this work, it has been possible to noticeably reduce the number of cases in which the compiler fails to detect buffer overflows in the Linux kernel, by reclassifying some types of arrays from dynamically sized to fixed sized. The additions to GCC can be used to compile any program, not just the Linux kernel.

These new GCC options are only the first step in a series of improvements that we implemented in the area of dynamically sized arrays. We’ll describe more of these efforts in subsequent blogs.

Array classes

Buffer overflows are a well known and frequently exploited software vulnerability. Buffer overflows can cause system crashes and provide bad actors with an entry point for cyber attacks, allowing attackers to gain unauthorized access to systems. The key to detecting a buffer overflow is to determine the size of the buffer. This task is quite tricky for certain types of arrays.

Let’s briefly review the various types of C arrays. There are two broad classes of arrays: fixed-sized arrays and dynamically-sized arrays. The former are arrays whose size is known and fixed at compile time. The compiler can easily check if accesses are within the bounds of these arrays, since the size is recorded in their declaration.

Dynamically-sized arrays include variable-length arrays (VLAs), flexible array member (FAMs) in structures, and pointer offsets (for example pointers into memory allocated via memory allocator calls such as malloc()). The size of these arrays is not known at compile time, and therefore determining when an access falls outside of the array is more complicated.

VLAs are arrays whose size depends on another variable whose value is known only at run time. VLAs are created on the stack and have a fixed size during their lifetime (but their size can vary for each instantiation). See below for examples of fixed size arrays, VLA and pointer offsets.

 

/* This is a fixed sized array */
char test[8] = "testing";

int do_something(int n)
{

   /* This is a VLA, whose size depends on the function parameter */
    int vla_array[n];

    for (int i = 0; i < n; ++i)
        vla_array[i] = i;

    return n+1;
}

#define NUM 10
int do_something_else ()
{
    /* p is an example of pointer offset */
    char *p = (char *) __builtin_malloc (sizeof (char) * NUM);
    for (int i = 0; i < NUM; i++)
      p[i] = i+1;
    return 0;
}

 

The more interesting class of arrays is Flexible Array Members (FAMs). FAMs are trailing arrays, meaning that they are the last element of a structure. Not all trailing arrays however are necessarily FAMs, because their size can be fixed and discoverable during compilation. The size of a FAM is not known until run time, after memory for the containing structure is allocated. It can be thought of as an incomplete array type. Determining the size of a FAM is often challenging.

Below is a set of examples of trailing arrays inside structures.

/* test is a flexible array member or trailing array */
struct flexible_array {
  int num;
  int test[0];
};

/* test is a flexible array member or trailing array */
struct flexible_array {
  int num;
  int test[1];
};

/* test is a flexible array member or trailing array */
struct flexible_array {
  int num;
  int test[];
};

/* test is a flexible array member or trailing array */
struct flexible_array {
  int num;
  int test[8];
};

 

Each of the alternatives shown in the example is valid C, but there are important caveats: though some of them use constants in their declarations and might look like fixed size arrays, due to historical reasons (which will be explained later) they are all considered to be FAM of unknown size at compile time. But are they really? This is the heart of the issue that we needed to address with our work in GCC. But before we dive into this, let’s finish our overview.

GCC provides two methods to detect and correct dangerous coding mistakes that can lead to buffer overflows (in addition to providing a minimal default level of protection):

 

  • GCC compiler options to enable array out-of-bound checking, including a compilation time warning option -Warray-bounds and a run-time sanitizer, -fsanitize=bounds;
  • GCC built-in functions to check object size (used directly in C code). They include a compilation time object size checker, __builtin_object_size (__bos), and a run-time object size checker, __builtin_dynamic_object_size (__bdos);

 

These tools work well in general. However, due to historic reasons, implementation issues, or lack of information, they are unable to handle some important cases. Our work addresses some of these situations.

The following table summarizes different array classes, whether the size information of the array is in its type declaration, and whether the GCC tools in GCC versions earlier than GCC 13 (that is before our work) can detect a buffer overflow for it.

Array Size Detect at Compilation Time? Detect at Run Time?
Class Example Declaration Is size in TYPE? Size -Warray-bounds __bos -fsanitize=bounds __bdos
fixed a[10] Y Constant Y Y Y Y
VLA a[n*4] Y Variable N N Y Y
FAM a[ ] N Unknown N N N N
pointer *a N Unknown N N N N

In the above table, the first and second columns list the different classes of arrays and the corresponding declaration examples; the third and the fourth columns show whether the size of the class of array is derivable from its type, and whether the size is constant or variable; the fifth and sixth columns show whether the current GCC compilation time tools, -Warray-bounds, and __builtin_object_size() (__bos), can detect a buffer overflow for the corresponding array; the last two columns show whether the current GCC run-time tools, -fsanitize=bounds, and __builtin_dynamic_object_size() (__bdos), can be used to detect a buffer overflow for the corresponding array.

Simple buffer overflow detection in GCC: Non FAM cases

A buffer overflow is typically introduced when an application fails to check that a reference into the buffer falls within the boundaries of the buffer. For instance, in C this could be due to not allocating enough buffer space for a string, or forgetting to reserve an element of the array for the string terminator character, or simply not checking against the size of the buffer. Such issues are particularly problematic in programming languages such as C and C++, since their language standards do not contain builtin buffer overflow protection.

Examples of out of bound accesses and uses of the available compiler options -Warray-bounds and -fsanitize=bounds to detect them are shown here. The first option operates at compile time, while the second triggers a run-time error:

$ cat array1.c
int a[10];
int* f(void)
{
  a[10] = 0; /* beyond end of array */
  return a;
}

$ gcc -O2 -Warray-bounds array1.c
array1.c: In function ‘f’:
array1.c:4:4: warning: array subscript 10 is above array bounds of ‘int[10]’ [-Warray-bounds=]
    4 |   a[10] = 0; /* beyond end of array */
      |   ~^~~~
array1.c:1:5: note: while referencing ‘a’
    1 | int a[10];
      |     ^
$ cat array2.c
#include <stdlib.h>

int foo (int k)
{
  struct S {
    int value1;
    int value2;
  } s[2];

  s[0].value1 = 0;
  s[1].value1 = 1;
  s[0].value2 = 10;
  s[1].value2 = 20;
  return s[k].value1;
}
int main ()
{
  return foo (2);
}

$ gcc -fsanitize=bounds array2.c
$ ./a.out
array2.c:14:11: runtime error: index 2 out of bounds for type 'S [2]'

A less obvious example of buffer overflow with fixed size array is shown below:

$ cat t1.c
#include <string.h>
char test[8] = "testing";

int main ()
{
 memset(test, 'b', 10); /* memset will write 10 bytes into the buffer "test"
                           whose length is 8 bytes, resulting in a buffer
                           overflow.  */
 __builtin_printf("Pass \n");
 return 0;
}

GCC detects this buffer overflow at compilation time by default, as shown below.

$ gcc t1.c
t1.c: In function ‘main’:
t2.c:6:2: warning: ‘memset’ writing 10 bytes into a region of size 8 overflows the destination [-Wstringop-overflow=]
    6 |  memset(test, 'b', 10); /* memset will write 10 bytes into the buffer "test"
      |  ^~~~~~~~~~~~~~~~~~~~~
t1.c:2:6: note: destination object ‘test’ of size 8
    2 | char test[8] = "testing";
      |      ^~~~

However, it is easy enough to defeat this simple check for out of bound access. With a slight modification to the above example, we get a different behavior:

$ cat t2.c
#include <string.h>
char test[8] = "testing";

void __attribute__ ((noinline)) my_memset (int n)
{
 memset(test, 'b', n); /* The 3rd argument now is "n", which is unknown
                          during compilation time since the routine "my_memset"
                          cannot be inlined.  */
 return;
}

int main ()
{
 my_memset(10);
 __builtin_printf("Pass \n");
 return 0;
}

$ gcc -O t2.c
$ ./a.out
Pass

In t2.c, the third argument n in the call to memset() is the same argument of the routine my_memset(). In this simple example we forced the my_memset() function to not be optimized by inlining, via the noinline attribute (if the function was inlined the variable n would be eliminated and set to 10, making the example pointless). As a result, the value of n remains unknown during compilation time and GCC fails to detect the buffer overflow both during the compilation and at run time.

 

Some help from GCC: Object size checking

In order to prevent more cases of out of bound access, such as the one above, GCC implements a limited buffer overflow protection mechanism. It determines the sizes of objects into which data is to be written and prevents the writes when the size is not sufficient. GCC issues warnings during compilation time if the sizes can be determined at that time. Otherwise it terminates the program at run time to prevent the buffer overflow. This is called object size checking. The core of this mechanism consists of two GCC built-in functions:

  • size_t __builtin_object_size (const void * ptr, int type): this function returns a constant number of bytes from ptr to the end of the object that the ptr pointer points to (if known at compile time).
  • size_t __builtin_dynamic_object_size (const void * ptr, int type): this function returns the number of bytes from ptr to the end of the object that the ptr pointer points to. The size returned may not be a constant.

In both functions, type is an integer constant from 0 to 3, which determines some of the function’s behavior.

In general, when the object size can be determined as a constant during compilation time, using __builtin_object_size() is sufficient.

If the object size can be determined as an expression, but not a constant during compilation time, one should use __builtin_dynamic_object_size(). The function __builtin_dynamic_object_size() covers the functionality of __builtin_object_size(). In other words __builtin_dynamic_object_size() could return a constant or return the result of an expression. Both built-in functions yield the best results if at least optimization level -O (i.e., -O1) is enabled.

To illustrate the usage and behavior of these functions let’s look at two examples:

$cat bdos_example.c
int __attribute__ ((noinline)) f(int n, int buf[n])
{
    /* This returns the size of buf. It is not known
       at compile time. It will return the result of
       an expression computed using the length of
       the array and the size of each element.*/
    return __builtin_dynamic_object_size(buf, 0);
}

int main()
{
    int n = 10;
    int buf[n];
    if (n * sizeof(int) != f(n, buf))
       __builtin_abort();
    else
       __builtin_printf("Pass \n");
    return 0;
}

and the second example, where the built-in function is not able to return a meaningful value:

$cat bos_example.c
int __attribute__ ((noinline)) f(int n, int buf[n])
{
    /* This call will not be able to return
       anything because buf is dynamic. */
    return __builtin_object_size(buf, 0);
}

int main()
{
    int n = 10;
    int buf[n];
    if (n * sizeof(int) != f(n, buf))
       __builtin_abort();
    else
       __builtin_printf("Pass \n");
    return 0;
}

Compiling these two files with:

$ gcc -O bdos_example.c -o bdos_example
$ gcc -O bos_example.c -o bos_example

Running the first executable we get:

$ ./bdos_example
Pass

In the first case the __builtin_dynamic_object_size(buf,0) will return the result of the expression n * 4. Running the second example we instead get:

$ ./bos_example
Aborted (core dumped)

In this second case __builtin_dynamic_object_size (buf, 0) has been replaced by __builtin_object_size(buf,0) which will detects that the object size is not a constant during compilation time and return -1 instead, leading to the program aborting.

The real benefit of these builtin functions is when they are used to harden many common string operation functions.

For example, in the case of the string operation function memset(), the built-in function __builtin___memset_chk() is provided to detect buffer overflow errors during compilation time or run time. Compared to the original function memset(), this built-in function has an additional last argument, which is the number of bytes remaining in the object the dest argument points to. For more details see: GCC Object Size Checking

The __builtin_object_size() or __builtin_dynamic_object_size() functions are usually called to compute the object size and, in this case, fed into the built-in function __builtin___memset_chk() as the last argument.

This is illustrated by the following:

#undef memset
#define memset(dest, src, n) \
 __builtin___memset_chk (dest, src, n, __builtin_object_size (dest, 1))

GCC provides similarly hardened built-in functions for memcpy(), mempcpy(), memmove(), memset(), strcpy(), stpcpy(), strncpy(), strcat(), strncat(), etc.

The usefulness of this approach can be shown by modifying the code in test case t2.c to use the hardened memset: now the buffer overflow can be detected during run time when the program is compiled (with optimization enabled):

$ cat t3.c
#include <string.h>
char test[8] = "testing";

#undef memset
#define memset(dest, src, n) \
 __builtin___memset_chk (dest, src, n, __builtin_object_size (dest, 1))

void __attribute__ ((noinline)) my_memset (int n)
{
memset(test, 'b', n);  /* Now the memset is hardened with
                          __builtin___memset_chk,
                          __builtin_object_size is called to determine the size
                          of the buffer that "test" points to, which is known to
                          be 8 bytes during compilation time.  */
 return;
}

int main ()
{
 my_memset(10);
 __builtin_printf("Pass \n");
 return 0;
}

$ gcc -O t3.c
$ ./a.out
*** buffer overflow detected ***: ./a.out terminated
Aborted (core dumped)

Object size checking doesn’t always help: The FAM case

Let’s focus now on FAMs to show how out of bound accesses for these types of arrays are impossible to detect. If we modify the previous example (t3.c) to change the array to be a trailing array in a structure, we’ll see that even though the size of this array might be known at compilation time, the __builtin_object_size() function cannot determine its size.

$ cat t4.c
#include <string.h>
struct trailing_array {
 int b;
 char test[8];
};

#undef memset
#define memset(dest, src, n) \
 __builtin___memset_chk (dest, src, n, __builtin_object_size (dest, 1))

void __attribute__ ((noinline)) my_memset (struct trailing_array *p, int n)
{
 memset(p->test, 'b', n);
 return;
}

int main ()
{
 struct trailing_array t_a;
 my_memset(&t_a, 10);
 __builtin_printf("Pass \n");
 return 0;
}

In t4.c, when we declare the array test[8] as a trailing array inside a struct, the buffer overflow of this array cannot be detected by GCC, either during the compilation, or at run time.

$ gcc -O t4.c
$ ./a.out
Pass

This is due to the compiler treating is as a FAM, even though it is a fixed size trailing array. In the next section we look into the reason why the compiler makes this apparently incorrect decision.

Why do such limitations with FAMs exist in GCC?

To explain the reason for this limitation, we need to understand the history of Flexible Array Members.

Flexible Array Members were formalized and added to the ISO C99 standard, in May 2000. C compilers, however, already supported FAMs according to different rules:

  • Any trailing array of a structure with 2 or more elements was treated as a FAM (with notation a[n]): This was the accepted C syntax before the introduction of the C90 standard.
  • Any trailing one-element array of a structure was treated as a FAM (with notation a[1]): In the ISO C90 standard, one-element arrays at the end of structures were used to represent FAMs by convention.
  • Any trailing zero-length array of a structure was treated as a FAM (with notation a[0]): This was (and still is) a valid GCC extension called zero-length array, introduced after C90, and used to support Flexible Array Members as an alternative to the previous notations. See more details here.
  • Any trailing array with no size is treated as FAM (with notation a[]): This notation is what the ISO C99 standard introduced.

The ISO C99 standard formalized the convention in a new and slightly different way. It allows to declare an array without a dimension and with no specified size (a[]), as long as the following conditions are satisfied:

  • The array should be inside a structure and be declared as the last member of the structure.
  • The structure must contain at least one more named member in addition to the flexible array member.

As a result, older C code, written before the introduction of FAMs into the ISO C99 standard could use any of the conventions (a[n], a[1], a[0]) listed above to represent FAMs, while newer C applications may also use the standard ISO C99 syntax (a[]) to represent FAMs. Therefore, for programs written after C99, there could be four different ways to declare FAMs. Another complication is that standard C++ does not include ISO C99 Flexible Array Members, forcing a C++ programmer to use either GCC’s zero-length array extension, or the one-element array convention (a[0] or a[1]).

 

There is more complexity around the matter of compilers handling trailing arrays, even those with a declared fixed size, such as test[8]. In particular, note the history of struct sockaddr in POSIX, which is a structure with a trailing array of apparently fixed size:

 

struct sockaddr {
 short int sa_family;
 char sa_data[14];
}

 

Unfortunately, the size of 14 cannot be interpreted as the actual fixed size of the array. This size is in reality determined by the type of the socket address family indicated by the first member of the structure. After the initial introduction of this component of the socket ABI, the classes of addresses in the networking world evolved, becoming more complex. The same data structure had to support addresses of many different types and sizes, which could well exceed 14 bytes.

This structure was part of a well established and vastly used ABI and it couldn’t be changed, forcing compilers to break the natural interpretation of this syntax as a fixed size trailing array. Compilers had no way of identifying via any heuristics that this structure and array declaration were special, because there was nothing particular about them. It became necessary to assume, to be safe, that all the trailing arrays in a structure were flexible arrays, for which the size cannot be known at compile time. For a more detailed account on this, please see the LWN article.

Before GCC 13 and our work, all the different variants of trailing arrays defined as a[n], a[1], a[0], a[] in legacy C code and non-standard C++ code, were treated as FAMs, due to this ambiguity, no matter what their sizes were: by default, all trailing arrays were treated as FAMs.

As a result, __builtin_object_size() would never be able to determine the size of a normal a[n] array at the end of a structure, since GCC (and other compilers) always assumed such array was a Flexible Array Member whose size is unknown.

The solution: -fstrict-flex-arrays

In order to resolve this issue, it is necessary to distinguish which trailing arrays are really FAMs from those that are not. The way to do this is to refine GCC’s interpretation of trailing arrays according to the four classes of declarations listed in the previous section, depending on user directives given via compilation options.

We added a new option -fstrict-flex-arrays=n and a corresponding attribute to GCC 13 and later. We also added a new warning option, -Wstrict-flex-arrays (See the manual here) to help programmers identify all the non C99 standard FAM usages.

  • Command line option: -fstrict-flex-arrays[=LEVEL] Treats the trailing array of a structure as a flexible array member in a stricter way depending on the specified level. The higher the level, the stricter the treatment of trailing arrays, and the fewer of them are considered FAMs.
  • Variable Attribute: strict_flex_arrays(LEVEL) The attribute should be attached to the trailing array field of a structure. It specifies the level of strictness when treating the trailing array field of a structure as a flexible array member.
  • Warning option: -Wstrict-flex-arrays: used together with -fstrict-flex-arrays[=LEVEL] enables warnings for different classes of trailing arrays. It has no effect and it is ignored if -fstrict-flex-arrays[=LEVEL] is not present.

The variable attribute can be used with, or without, the option -fstrict-flex-arrays. If both the attribute and the option are present, the level of the strictness for the specific trailing array field is determined by the attribute used in the code.

The value of LEVEL controls the strictness.

There are four different values for LEVEL:

  • LEVEL=0 is the least strict level. All trailing arrays of structures are treated as flexible array members. If the option is not present, LEVEL=0 is the default.
  • LEVEL=1, the trailing array is treated as a flexible array member if it is declared as either a[], a[0], or a[1];
  • LEVEL=2, the trailing array is treated as a flexible array member if it is declared as either a[], or a[0].
  • LEVEL=3 is the strictest level. Only if the trailing array is declared as a flexible array member per the ISO C99 standard and onward (a[]), it is treated as a flexible array member. It is the default if the option is present without a value.

Levels 1 and 2 are provided to support older applications that use GCC’s zero-length array (a[0]), or the one-size array as flexible array member (a[1]).

With -fstrict-flex-arrays=3, GCC only treats a[] as FAM, all other trailing arrays are treated as normal arrays whose size is known. Object size checking in GCC works correctly with these normal trailing arrays.

Let’s take the previous example t4.c, to see the benefits of having this new GCC options. If the -fstrict-flex-arrays=3 option is added to the compilation line, the buffer overflow is detected successfully during run time:

$ gcc -O -fstrict-flex-arrays=3 t4.c
$ ./a.out
*** buffer overflow detected ***: ./a.out terminated

This can also be detected if less strict, level 2, or level 1, options are used. If instead the level 0 option is used the overflow is not detected.

Guidance to users

Let’s look at a more complex example that will be used to show the behavior of the various options given to GCC during compilation, by using different levels of strictness and seeing what their effects are on the four categories of FAMs we discussed:

$ cat trailingarrays.c
#include <stdio.h>
#include <stdlib.h>

struct trailing_array_1 {
    int a;
    int c[4];
};

struct trailing_array_2 {
    int count;
    int c[1];
};

struct trailing_array_3 {
    int count;
    int c[0];
};

struct trailing_array_4 {
    int count;
    int c[];
};

void __attribute__((__noinline__)) stuff(
    struct trailing_array_1 *normal,
    struct trailing_array_2 *trailing_1,
    struct trailing_array_3 *trailing_0,
    struct trailing_array_4 *trailing_flex)
{
    /* These would produce out of bound errors
       if these array are not considered FAMs */

    normal->c[5] = 5;
    trailing_1->c[2] = 2;
    trailing_0->c[1] = 1;
    trailing_flex->c[10] = 10;
}

int main()
{
  struct trailing_array_1 normal_trailing_array;
  struct trailing_array_2 *one_in_struct;
  struct trailing_array_3 *zero_in_struct;
  struct trailing_array_4 *fam_in_struct;

  int fam_length = 6;
  int zero_length = 6;
  int one_length = 6;


  /* These allocate memory for each structure
     in each case of trailing arrays (except the first) */

  one_in_struct = (struct trailing_array_2 *)
         malloc (sizeof (struct trailing_array_2) + one_length);
  one_in_struct->count = one_length;

  zero_in_struct = (struct trailing_array_3 *)
         malloc (sizeof (struct trailing_array_3) + zero_length);
  zero_in_struct->count = zero_length;

  fam_in_struct = (struct trailing_array_4 *)
         malloc (sizeof (struct trailing_array_4) + fam_length);
  fam_in_struct->count = fam_length;


  /* Size of each structure, this includes the size
     of the first member of the structure. Which is
     an int of 4 bytes (in our hardware) */

  /* The first struct is 20 bytes */
  printf ("normal_trailing_array struct size is %d\n", sizeof (normal_trailing_array));

  /* The second struct is 8 bytes */
  printf ("one_in_struct size is %d\n", sizeof (*one_in_struct));

  /* The third struct is 4 bytes */
  printf ("zero_in_struct size is %d\n", sizeof (*zero_in_struct));

  /* The fourth struct is 4 bytes */
  printf ("fam_in_struct size is %d\n", sizeof (*fam_in_struct));

  stuff (&normal_trailing_array,
         one_in_struct,
         zero_in_struct,
         fam_in_struct);
  return 0;
}

The above example uses all the possible ways to define a FAM. However, if we build with different levels of the -fstrict-flex-arrays option, a different subset of the four trailing arrays is reported as non-conformant.

This can be seen below with the strictest version of the new compilation option and by enabling the warning, -Wstrict-flex-arrays,-fstrict-flex-arrays=3, the compiler is forced to only accept the C99 definition as valid, while issuing warnings for the other ones.

$ gcc -O2 -Wstrict-flex-arrays -fstrict-flex-arrays=3 trailingarrays.c
trailingarrays.c: In function ‘stuff’:
trailingarrays.c:33:14: warning: trailing array ‘int[4]’ should not be used as a flexible array member [-Wstrict-flex-arrays]
   33 |     normal->c[5] = 5;
      |     ~~~~~~~~~^~~
trailingarrays.c:6:9: note: while referencing ‘c’
    6 |     int c[4];
      |         ^
trailingarrays.c:34:18: warning: trailing array ‘int[1]’ should not be used as a flexible array member [-Wstrict-flex-arrays]
   34 |     trailing_1->c[2] = 2;
      |     ~~~~~~~~~~~~~^~~
trailingarrays.c:11:9: note: while referencing ‘c’
   11 |     int c[1];
      |         ^
trailingarrays.c:35:18: warning: trailing array ‘int[0]’ should not be used as a flexible array member [-Wstrict-flex-arrays]
   35 |     trailing_0->c[1] = 1;
      |     ~~~~~~~~~~~~~^~~
trailingarrays.c:16:9: note: while referencing ‘c’
   16 |     int c[0];
      |         ^

With LEVEL=2 and the warning enabled, it will only accept the [0] and [] syntax:

$ gcc -O2 -Wstrict-flex-arrays -fstrict-flex-arrays=2 trailingarrays.c
trailingarrays.c: In function ‘stuff’:
trailingarrays.c:33:14: warning: trailing array ‘int[4]’ should not be used as a flexible array member [-Wstrict-flex-arrays]
   33 |     normal->c[5] = 5;
      |     ~~~~~~~~~^~~
trailingarrays.c:6:9: note: while referencing ‘c’
    6 |     int c[4];
      |         ^
trailingarrays.c:34:18: warning: trailing array ‘int[1]’ should not be used as a flexible array member [-Wstrict-flex-arrays]
   34 |     trailing_1->c[2] = 2;
      |     ~~~~~~~~~~~~~^~~
trailingarrays.c:11:9: note: while referencing ‘c’
   11 |     int c[1];
      |         ^

And finally with LEVEL=1 and the warning enabled, it will accept the [1], [0] and [] syntax, but not the [n] syntax

 
$ gcc -O2 -Wstrict-flex-arrays -fstrict-flex-arrays=1 trailingarrays.c
trailingarrays.c: In function ‘stuff’:
trailingarrays.c:33:14: warning: trailing array ‘int[4]’ should not be used as a flexible array member [-Wstrict-flex-arrays]
   33 |     normal->c[5] = 5;
      |     ~~~~~~~~~^~~
trailingarrays.c:6:9: note: while referencing ‘c’
    6 |     int c[4];
      |         ^

Using LEVEL=0 will not produce any warnings.

As previously mentioned, if the code base is relatively old, there are very good chances that it uses all the four variants of FAMs, some of which are not actually FAMs, but just regular fixed size trailing arrays. GCC can now distinguish among the four cases, and the code can be made more secure thanks to this new feature.

First it is necessary to explore the code for these occurrences using the -Wstrict-flex-arrays option (See man page) coupled with -fstrict-flex-arrays=LEVEL cycling through the various values for LEVEL. After the usages of a[0], a[1], a[n] have been identified, they can be converted to proper FAMs as necessary. This must be done very carefully because other code could depend on assumptions about the layout and size of these arrays. Also, some of them might be intended to be real fixed-size trailing arrays.

Also note that eliminating the use of the old style FAM usage (also called fake FAMs) will possibly remove other subtle errors in the code. For instance consider that the function sizeof() returns values that are somewhat counter-intuitive for these type of trailing arrays. Operations using such values, for instance to check array bounds for writing, would produce wrong results, potentially overwriting memory.

In the previous example trailingarrays.c the size of each of the four trailing arrays computed using sizeof() would be:

  • 16 bytes for c[4] in trailing_array_1,
  • 4 bytes for c[1] in trailing_array_2,
  • 0 bytes for c[0] in trailing_array_3,
  • for c[] in trailing_array_4 the compiler generates an error:

     

    trailingarrays.c:44:41: error: invalid application of ‘sizeof’ to incomplete type ‘int[]’
       44 |   printf ("four.c size is %d\n", sizeof (four->c));
          |                                                ^
    

At the end of this process all the identified occurrences of a[0], a[1], a[n] that are to be used as real FAMs in the program, will be replaced by the C99 syntax a[], and treated like real FAMs.

Now the code is ready to be fully compiled with the -fstrict-flex-arrays=3 option to assure that only the a[] syntax is treated as FAM (this corresponds to the smallest possible subset of these trailing arrays). In addition compile with the -Warray-bounds option to enable out of bound accesses on fixed sized arrays and non-FAM trailing arrays.

Alternatively the code can be compiled with -Wstrict-flex-arrays warnings enabled, together with -fstrict-flex-arrays=3 (as shown above), and the compiler will generate different warnings but would still identify the non FAM array uses.

If the code is allowed to crash or abort, the other GCC option -fsanitize=bounds could also be used to maximize the errors that can be caught.

Let’s look at the trailingarrays.c example again to understand the type of protection that combinations of these compiler options offer.

  • Using gcc -O2 -Wstrict-flex-arrays -fstrict-flex-arrays=3 produces compilation warnings like the ones above.
  • Using gcc -O2 -Warray-bounds -fstrict-flex-arrays=3 produces the following compilation warnings detecting the out of bounds accesses for the non FAMs cases:
    trailingarrays.c: In function ‘stuff’:
    trailingarrays.c:33:14: warning: array subscript 5 is above array bounds of ‘int[4]’ [-Warray-bounds=]
       33 |     normal->c[5] = 5;
          |     ~~~~~~~~~^~~
    trailingarrays.c:6:9: note: while referencing ‘c’
        6 |     int c[4];
          |         ^
    trailingarrays.c:34:18: warning: array subscript 2 is above array bounds of ‘int[1]’ [-Warray-bounds=]
       34 |     trailing_1->c[2] = 2;
          |     ~~~~~~~~~~~~~^~~
    trailingarrays.c:11:9: note: while referencing ‘c’
       11 |     int c[1];
          |         ^
    trailingarrays.c:35:18: warning: array subscript 1 is outside array bounds of ‘int[0]’ [-Warray-bounds=]
       35 |     trailing_0->c[1] = 1;
          |     ~~~~~~~~~~~~~^~~
    trailingarrays.c:16:9: note: while referencing ‘c’
       16 |     int c[0];
          |         ^
    
  • Using gcc -02 -fsanitize=bounds -fstrict-flex-array=3 will compile fine, and produce these run-time warnings, detecting the out of bound accesses as the case above.
    trailingarrays.c:33:14: runtime error: index 5 out of bounds for type 'int [4]'
    trailingarrays.c:34:18: runtime error: index 2 out of bounds for type 'int [1]'
    trailingarrays.c:35:18: runtime error: index 1 out of bounds for type 'int [*]'
    	  

 

Changing the level set for -fstrict-flex-arrays changes how many of these four forms of trailing arrays are considered FAMs and therefore how many of them will generate warnings and errors. The stricter the level, the more out of bound accesses will be caught.

Note that for real FAMs, as defined by C99, there is no way to compute their size and, therefore, determine if there is an out of bound access. This will be addressed by future work to be presented in a future blog.

In summary, if the usages of non C99 standard notation for FAMs as trailing arrays (a[n], a[1], or a[0]) are completely eliminated from the source code, GCC is able to detect more buffer overflow errors in the source code for the non FAM cases, because it will be able to check the operations against the size of these arrays (which are not considered FAMs anymore), producing more secure code.

Real world examples

The Linux kernel is one of the major consumers of this new security feature. In fact, it is also the one that initially asked for this feature back in 2021 in order to improve its buffer overflow detection. As mentioned earlier, this was done as part of the larger project called KSPP, to generally improve the level of safety on the Linux kernel through stricter coding rules.

Since 2020, the Linux kernel started the work to eliminate all the non-standard FAM usages. It took over 3 years to eliminate around 600 occurrences of such usages in the Linux kernel. (See the long list of commits)

After the -fstrict-flex-arrays was available in GCC 13, the Linux kernel was ready to enable -fstrict-flex-arrays=3 by default starting from v6.5 in 2023 (see the commit).

In the Linux kernel Makefile we see:

# To gain proper coverage for CONFIG_UBSAN_BOUNDS and CONFIG_FORTIFY_SOURCE,
# the kernel uses only C99 flexible arrays for dynamically sized trailing
# arrays. Enforce this for everything that may examine structure sizes and
# perform bounds checking.
KBUILD_CFLAGS += $(call cc-option, -fstrict-flex-arrays=3)

A detailed description of the work done in the kernel to enable -fstrict-flex-arrays can be found here.

In addition to the Linux kernel, many other applications, for example, systemd, oauth-tool, diff-utils, man-db, coreutils, and gnulib all use -fstrict-flex-arrays too (on Debian distributions you can search using Codesearch).

Conclusion

In this blog we have presented new options for the GCC compiler that greatly help to make applications more secure.

The benefit from these features is optimal if the code is restructured to deal with legacy constructs that have been introduced prior to C99. These basically legalized certain coding practices, but also pose a potential security risk.

The new options can be used to help identify such legacy cases and to make sure they are not accidentally reintroduced in the code base. We recommend to use the warnings to clean up the code and from there on leverage the benefits of the additional checks presented here.

Acknowledgements

Qing Zhao is the author of this work. She would like to thank Kees Cook, Siddhesh Poyarekar, Martin Sebor, Jakub Jelinek, James Y Knight for their insights while working on this GCC feature.

Some further ways developers can secure their code

GCC provides a rich set of options and tools to aid developers wanting to better secure their C and C++ code.

These two blogs describe some additional GCC options and tools that developers could utilize in order to make their code more secure: