X

Improve application security, performance, and scalability

Recent Posts

Dear Raj, SSM Has Keys. IBM Tells Me They Got Keys Too. True?
- Confused in FUD City

Dear Confused in FUD City,Good question. You will be relieved to know that I have been asked this questions many times, so you are not alone.Let me start by saying that only thing that is common between some explanations of SSM (Silicon Secured Memory) and IBM's storage-protection keys is the word key. After all, the English language has only so many words.IBM has had storage protection keys for decades. (Yes, decades!. That alone should tell us that it is unlikely to be on the same level as SSM). The protection provided by IBM's storage-protection keys is analogous to memory protection offered across multiple processes in a system. Within a complex software program, IBM storage protection keys provide a mechanism to keep memory pages of one (logical) module safe from another. A module can set up a memory area with a key, say 4. If another module declares itself to be operating with access to key 4 area, it can do so, otherwise not. Note that the granularity of this feature is a memory page, which are of the sizes 4KB, 64KB, 16MB, etc.This is good and useful, However, Code 1 can be safe from Code 2, but it is not safe from itself. So, it cannot protect you from vulnerabilities, such as, infamous recent ones, Heartbleed and Venom.Let's look at what a memory page looks like up close.The memory pages are not monolithic blobs. They contain several pieces of distinct information, often called buffers. The deadliest vulnerabilities occur when (carelessly programmed) access to a buffer overflows (or underflows) into a neighboring buffer. The Heartbleed vulnerability was a case of a buffer overflow read and Venom a case of a buffer overflow write.SSM allows easy mechanism to color these buffers distinctly. You can read the details in my blog posts on SSM or in the merged white paper.Theoretically, you could use IBM's storage-protection keys to color buffers, but, remember, it operates on the granularity of a memory page. So for every buffer of say 64 bytes or 128 bytes, or even 1K bytes, you will be using up 4K to multiple mega bytes of memory. So, it is just not practical. That is not all, to protect buffers from each other, the program would need to set its keys for EVERY access to any of its buffers, so that it cannot overflow into the next. It just wouldn't be a meaningful endeavor.

Dear Confused in FUD City, Good question. You will be relieved to know that I have been asked this questions many times, so you are not alone.Let me start by saying that only thing that is common...

Silicon Secured Memory - It's Better Than You Might Think

Robust Checking from Smart Algorithms and ProbabilitySilicon Secured Memory (SSM) uses a subset of the bits in the pointer, and it still provides very robust error detection. Using a single bit we get a 50% chance of a pointer matching a region of memory when it shouldn’t. If we were to use two bits we would have a 75% chance of catching an error. With three bits we would have 87.5% chance, and so on.However, that is true only if the colors were assigned randomly. The memory allocation routines give different colors to adjacent areas. Therefore, a buffer overflow into the neighboring area is detected 100% of the time. The security vulnerabilities caused by buffer overflows, such as, Heartbleed and Venom, are stopped every time. Freed memory access is also caught reliably.What's more, even stale pointer access (a freed memory access to an area that has been subsequently allocated for another purpose), which no software tool to date detects, is also caught nearly 100% of the time That is because in practice a stale pointer access happens very soon after a reuse of an area, and the area will have been assigned a new color on its reuse. The area does not return to the original color until the memory management routines cycle through many allocations and free of the same area.

Robust Checking from Smart Algorithms and Probability Silicon Secured Memory (SSM) uses a subset of the bits in the pointer, and it still provides very robust error detection. Using a single bit we get...

Silicon Secured Memory in Action!

Detecting Memory Access Errors using Hardware SupportHere is an example program that shows the hardware detecting four types of errors – buffer overflow, freed memory access, stale pointer access, and freeing a memory more than once. 1 #include <stdlib.h> 2 #include <stdio.h> 3 int main() { 4 int *area1 = malloc(sizeof(int)*16); 5 int *area2 = malloc(sizeof(int)*100); 6 7 for (int i = 0; i <= 16; i++) 8 area1[i] = 0; // Array Out of Bounds 9 10 free(area1); 11 area1[0] = 0; // Freed Memory Access 12 13 char *area3 = malloc(sizeof(char)*64); 14 if ((void *)area1 == (void *)area3) 15 printf("New area3 is same as old area1\n"); 16 area1[0] = 0; // Stale Pointer Access 17 18 free(area3); 19 free(area3); 20 21 return 0; 22 }This program can be run under discover using Silicon Secured Memory (SSM, previously know as ADI) by passing the flag "-i adi" to discover:$ cc t.c -g -m64$ discover -i adi -w – a.out$ a.outThe first problem that SSM detects is the buffer overflow on line 8, where the array area1 is accessed with index 16 while the highest valid index is 15.Discover indicates the point in the code where the access occurs, plus the location in the code where the buffer is allocated.ERROR 1 (ABW): writing to memory beyond array bounds at address 0x200000021047e040: main() + 0x38 <t.c:8> 5: int *area2 = malloc(sizeof(int)*100); 6: 7: for (int i = 0; i <= 16; i++) 8:=> area1[i] = 0; // Array Out of Bounds 9: 10: free(area1); 11: area1[0] = 0; // Freed Memory Access _start() + 0x108 was allocated at (64 bytes): main() + 0x8 <t.c:4> 1: #include 2: #include 3: int main() { 4:=> int *area1 = malloc(sizeof(int)*16); 5: int *area2 = malloc(sizeof(int)*100); 6: 7: for (int i = 0; i <= 16; i++) _start() + 0x108 The next problem detected is the write to freed memory at line 11, the memory was freed earlier at line 10.ERROR 2 (FMW): writing to freed memory at address 0x200000021047e000: main() + 0x6c <t.c:11> 8: area1[i] = 0; // Array Out of Bounds 9: 10: free(area1); 11:=> area1[0] = 0; // Freed Memory Access 12: 13: char *area3 = malloc(sizeof(char)*64); 14: if ((void *)area1 == (void *)area3) _start() + 0x108 was allocated at (64 bytes): main() + 0x8 <t.c:4> 1: #include 2: #include 3: int main() { 4:=> int *area1 = malloc(sizeof(int)*16); 5: int *area2 = malloc(sizeof(int)*100); 6: 7: for (int i = 0; i <= 16; i++) _start() + 0x108 freed at: main() + 0x5c <t.c:10> 7: for (int i = 0; i <= 16; i++) 8: area1[i] = 0; // Array Out of Bounds 9: 10:=> free(area1); 11: area1[0] = 0; // Freed Memory Access 12: 13: char *area3 = malloc(sizeof(char)*64); _start() + 0x108 There is a stale pointer access at line 16. The memory pointed to by area1 was freed at line 10, but the memory was reused for area3 at line 13. discover reports this as a write to freed memory – even though the memory has been repurposed. This is an example of the kind of error that it is very hard for a software only solution to detect.ERROR 3 (FMW): writing to freed memory at address 0x200000021047e000: main() + 0xb4 <t.c:16> 13: char *area3 = malloc(sizeof(char)*64); 14: if ((void *)area1 == (void *)area3) 15: printf("New area3 is same as old area1\n"); 16:=> area1[0] = 0; // Stale pointer access 17: 18: free(area3); 19: free(area3); _start() + 0x108 was allocated at (64 bytes): main() + 0x8 <t.c:4> 1: #include <stdlib.h> 2: #include <stdio.h> 3: int main() { 4:=> int *area1 = malloc(sizeof(int)*16); 5: int *area2 = malloc(sizeof(int)*100); 6: 7: for (int i = 0; i <= 16; i++) _start() + 0x108 freed at: main() + 0x5c 7: for (int i = 0; i <= 16; i++) 8: area1[i] = 0; // Array Out of Bounds 9: 10:=> free(area1); 11: area1[0] = 0; // Freed Memory Access 12: 13: char *area3 = malloc(sizeof(char)*64); _start() + 0x108 The final error reported by discover is the double free of area3 at lines 18 and 19.ERROR 4 (DFM): double freeing memory at address 0x300000021047e000: main() + 0xc8 <t.c:19> 16: area1[0] = 0; // Stale pointer access 17: 18: free(area3); 19:=> free(area3); 20: 21: return 0; 22: } _start() + 0x108 was allocated at (64 bytes): main() + 0x74 <t.c:13> 10: free(area1); 11: area1[0] = 0; // Freed Memory Access 12: 13:=> char *area3 = malloc(sizeof(char)*64); 14: if ((void *)area1 == (void *)area3) 15: printf("New area3 is same as old area1\n"); 16: area1[0] = 0; // Stale pointer access _start() + 0x108 freed at: main() + 0xbc <t.c:18> 15: printf("New area3 is same as old area1\n"); 16: area1[0] = 0; // Stale pointer access 17: 18:=> free(area3); 19: free(area3); 20: 21: return 0; _start() + 0x108 DISCOVER SUMMARY: unique errors : 4 (4 total)

Detecting Memory Access Errors using Hardware Support Here is an example program that shows the hardware detecting four types of errors – buffer overflow, freed memory access, stale pointer access, and...

Surprise! Unexpected Benefits of Hardware Support for Detection of Memory Access Errors

The obvious advantage to hardware support for memory error detection is the massive performance advantage. However, there is another advantage to Silicon Secured Memory (SSM) that is not immediately apparent. SSM can pick up a range of errors that normal instrumentation cannot identify. Earlier we discussed that typical software instrumentation has a problem when there are stale pointers to reassigned memory locations, or if a pointer happens to point to a valid memory location. It is very hard for software to be able to handle these situations because it has no idea whether the memory location is a valid one for the pointer to address. SSM on the other hand encodes the “color” of a memory location into both the pointer and the memory location. So only a “red” pointer can address a “red” memory location.Let’s consider how this changes the stale pointer situation. When a memory location is freed, the call to free() can change the color of that block of memory. So now a stale pointer to that block of memory will be the wrong color to access it. Using this approach we can detect an access to that block through a freed pointer.Now imagine that the block of memory is returned for new use by another call to malloc(). In this situation we can change the color of the block again. An access through the stale pointer continues to report an error by trapping.Here is the example in C.int *area1 = malloc(64); free(area1); char *area2 = malloc(64); // area2 gets the memory area just freed by area1area1[0] = 0; // Stale Pointer Access The other situation is where a pointer ends up with random data that is dereferenced, and happens to point to a valid block of data. In this case the pointer is likely to be of the wrong type, so the error will be detected.All these situations are very hard to detect in software, but are caught with hardware support. So the hardware support is not only faster, and easier to apply to an existing application, it also identifies an additional range of problems.

The obvious advantage to hardware support for memory error detection is the massive performance advantage. However, there is another advantage to Silicon Secured Memory (SSM) that is not immediately...

The Holy Grail - Real Time Memory Access Checking

Limitations of dynamic instrumentationThe most obvious limitation of dynamic instrumentation with tools, such as, Valgrind and Purify, is that the instrumented program runs much slower than the original program. This is because the instrumented program needs to keep a database of each allocated chunk of memory. Each memory read and write instruction has to be augmented with other instructions to read the information from the database to find out whether the memory location is valid. The instrumented programs typically run more than 30x slower.This slowdown often makes it impractical to do dynamic checking of a program with a large test suite. Consequently, dynamic checking is mostly used for debugging a specific problem or with a smaller test suite.There are also some limitations on the errors that dynamic instrumentation can detect. The ability to detect memory errors relies on being able to distinguish between valid memory accesses and invalid memory accesses. Some kinds of memory access are clearly an error, for example a read of uninitialized memory is unambiguously a problem. Other situations are not so clear cut.As an example consider an application that allocates a structure, uses that structure and then frees it. If a pointer to this region of memory is used, then the tool can detect that it is an access to invalid memory, and can report it as a freed memory access error. This is known as a dangling pointer – a pointer that points to a region of memory that is no longer valid.However, if a later malloc() reuses the same region of memory, then the memory is considered valid again. Now a memory access through the stale pointer is indistinguishable from a memory access through a legitimate pointer to the region. So it is not possible for a tool to report an error when the stale pointer is used.int *area1 = malloc(64); free(area1); char *area2 = malloc(64); // area2 gets the memory area just freed by area1area1[0] = 0; // Stale Pointer Access There is a similar situation where a pointer gets corrupted. If the corrupted pointer happens to point to a valid region of memory, then it’s not possible for a tool to determine that this is a corrupted, rather than legitimate, pointer.Hardware support for memory error detectionNew SPARC processors, starting with the SPARC M7 processor, have hardware support for Silicon Secured Memory (SSM, previously known as ADI). This hardware support allows real-time detection of memory access errors.Data is stored in memory in units of 64 bytes called cache lines. So when a data of one or more bytes is loaded from memory, the entire block of 64 bytes containing that data is fetched. The latest SPARC processors extend this by adding four additional bits to each cache line. Fetching the 64 byte cache line also fetches these additional bits. These four bits are invisible to the application and are used to hold additional information for SSM.The best way of thinking of the bits is to imagine them containing a color. For example, a value of one could be thought of as Red, a value of two Green, and so on. So a cache line, of 64 bytes, can be thought of as both containing 64 bytes of data and having a color.Whenever we need to access a memory location, we need to have a pointer to that memory location. Pointers are 64 bits in size which allows a 64-bit processor to potentially access 16 EiB of data – about 17,000,000 TB of data. There’s no current systems that can hold this much memory. For example, a SPARC M7-32 system can contain a staggering 64 TB of memory. Consequently a 64-bit processor does not need to use all the 64-bits in a pointer. Normally the unused bits are constrained to be all zero or all one, but SSM uses them for a different purpose.Instead of requiring the most significant four bits to be all zero or all one, SSM uses them to store color values. This means all the pointers can be thought of as colored in the same way as all the cache lines in memory are colored.SSM uses the fact that we can color both pointer and memory to check for invalid memory accesses. A “green” cache line can only be accessed through a “green” pointer. It is an error to use a “green” pointer to access a “red” cache line. The hardware will cause a trap when this color mismatch occurs.The advantages of hardware supportThe most obvious advantage to hardware support for memory error detection is the massive performance advantage. The hardware takes responsibility for checking that every memory access is valid, and this usually only incurs a cost if the access is invalid and the hardware has to trap to report the error. Consequently most applications run at close to their usual speeds.Another important advantage is that the software changes needed to support SSM can be provided in a library. An application does not need to have any instrumentation added in order for it to be checked. This means even existing applications where the source code has been lost can be checked for correctness. For example, if the application is run with command a.out, the following will enable SSM checking of the application.% LD_PRELOAD_64=/lib/compilers/sparcv9/libdiscoverADI.so a.out

Limitations of dynamic instrumentation The most obvious limitation of dynamic instrumentation with tools, such as, Valgrind and Purify, is that the instrumented program runs much slower than the...

Solving Trickier Problems - Detecting Dynamic Memory Access Errors Using Discover

Although a subset of memory access errors can be detected at compile time, the majority can only be detected at runtime. Some memory error conditions, like double free, can be detected using a light veneer of functionality over the existing library functions. Other memory access errors can only be detected through a deep instrumentation of the target application.Studio provides the tool discover to do both light and deep analysis of memory access errors in a target application. Light instrumentation of library calls has a very small impact on the performance of an application, most applications continue to run at essentially normal speeds. The weakness of this level of analysis is that it only catches problems at the function call level. Consider the following code:#include <stdlib.h>void main(){ char * string = (char*)malloc(1024); free(string); free(string);}This code has a double free() of allocated memory. Of course, given that both calls are in the same scope a static check of the code would also pick up the error. However, most errors of this kind occur in more complex code sequences.Double free is one of the errors that can be picked up with a light instrumentation of the applications library calls. To perform a light instrumentation we compile the binary, then process the resulting binary with the tool discover, this produces an instrumented version of the application which can then be run. If the -l command line option is passed to discover then it only instruments the library calls. The results of this instrumentation are:$ cc -g double.c$ discover -l -w - ./a.out$ ./a.outERROR 1 (DFM): double freeing memory "*string" at address 0x40560 at: main() + 0x20 <double.c:7> 4: { 5: char * string = (char*)malloc(1024); 6: free(string); 7:=> free(string); 8: } 9: _start() + 0x108 was allocated at (1024 bytes): main() + 0x4 <double.c:5> 2: 3: void main() 4: { 5:=> char * string = (char*)malloc(1024); 6: free(string); 7: free(string); 8: } _start() + 0x108 freed at: main() + 0x14 <double.c:6> 3: void main() 4: { 5: char * string = (char*)malloc(1024); 6:=> free(string); 7: free(string); 8: } 9: _start() + 0x108***************** Discover Memory Report *****************No allocated memory left on program exit.DISCOVER SUMMARY: unique errors : 1 (1 total) unique warnings : 0 (0 total)The report produced for the double free indicates where the second free occurred. It also helpfully points out where the memory was allocated, and where it was first freed. This should enable most developers to reconstruct the program flow that lead to the double free error.In the example we passed the option “-w -” to discover. This tells discover to produce a report to stderr. Without this, the default is to produce an html report that can be examined in a web browser.If the application has been compiled with Solaris Studio, then discover can do a far deeper analysis of memory access errors. The tool actually adds instrumentation code into every memory operation, and this instrumentation checks whether the memory access is valid or not. The downside of this degree of instrumentation is that it can have considerable impact on runtime.Uninitialized memoryAs an example of this consider the earlier code example with accessed uninitialized memory. Using discover we get a report that shows the location of the invalid memory access and the call stack that got us to that point.$ cc -g uninit.c$ discover -w - ./a.out$ ./a.outERROR 1 (UMR): accessing uninitialized data at address 0xffbffba4 (4 bytes) on the stackat: printint() + 0xa8 <uninit.c:5> 2: 3: void printint(int *i) 4: { 5:=> printf("Integer = %i\n",*i); 6: } 7: 8: void main() main() + 0x20 <uninit.c:11> 8: void main() 9: { 10: int i; 11:=> printint(&i); 12: } 13: _start() + 0x108***************** Discover Memory Report *****************No allocated memory left on program exit.DISCOVER SUMMARY: unique errors : 1 (1 total) unique warnings : 0 (0 total)Access past array boundsEarlier we showed an example where the code accessed beyond the end of an array.Discover can diagnose using deep memory access checking, as shown in the example below.-bash-4.1$ cc -g array.c-bash-4.1$ discover -w - ./a.out-bash-4.1$ ./a.outERROR 1 (ABR): reading memory beyond array bounds at address 0x40590 (4 bytes) on theheap at: printarray() + 0x144 <array.c:8> 5: { 6: for (int i=0; i<len; i++) 7: { 8:=> printf("Index %i = %i\n", i, array[i]); 9: } 10: } 11: main() + 0x74 <array.c:15> 12: void main() 13: { 14: int * array = (int*) calloc(sizeof(int), 10); 15:=> printarray(array,11); 16: } 17: 18: _start() + 0x108 was allocated at (40 bytes): main() + 0x20 <array.c:14> 11: 12: void main() 13: { 14:=> int * array = (int*) calloc(sizeof(int), 10); 15: printarray(array,11); 16: } 17: _start() + 0x108***************** Discover Memory Report *****************1 allocation at 1 location left on the heap with a total size of 40 bytes LEAK 1: 1 allocation with total size of 40 bytes main() + 0x20 <array.c:14> 11: 12: void main() 13: { 14:=> int * array = (int*) calloc(sizeof(int), 10); 15: printarray(array,11); 16: } 17: _start() + 0x108DISCOVER SUMMARY: unique errors : 1 (1 total) unique warnings : 0 (0 total)Discover is also able to report memory leaks. In this instance discover is reporting that 40 bytes were left on the heap. Access to previously freed memoryInstrumentation can easily detect situations where there is an access to previously freed memory. Since discover instruments the binary it can point out the exact place in the code where the previously freed memory is accessed, and it can indicate where the memory was allocated, and where it was freed. -bash-4.1$ cc -g freed.c-bash-4.1$ discover -w - ./a.out-bash-4.1$ ./a.outERROR 1 (FMR): reading from freed memory at address 0x50010 (4 bytes) on the heap at: printint() + 0xa8 <freed.c:6> 3: 4: void printint(int *i) 5: { 6:=> printf("Integer = %i\n", *i); 7: } 8: 9: void main() main() + 0x78 <freed.c:13> 10: { 11: int * i = malloc(sizeof(int)); 12: free(i); 13:=> printint(i); 14: } 15: _start() + 0x108 was allocated at (4 bytes): main() + 0x1c <freed.c:11> 8: 9: void main() 10: { 11:=> int * i = malloc(sizeof(int)); 12: free(i); 13: printint(i); 14: } _start() + 0x108 freed at: main() + 0x6c <freed.c:12> 9: void main() 10: { 11: int * i = malloc(sizeof(int)); 12:=> free(i); 13: printint(i); 14: } 15: _start() + 0x108Integer = 327704***************** Discover Memory Report *****************No allocated memory left on program exit.DISCOVER SUMMARY: unique errors : 1 (1 total) unique warnings : 0 (0 total)

Although a subset of memory access errors can be detected at compile time, the majority can only be detected at runtime. Some memory error conditions, like double free, can be detected using a light...

Let's Get The Low Hanging Fruits - Static detection of memory access errors using Previse

There is a subset of memory access errors that can be detected at compile time. This is obviously the ideal time to warn about these issues because the developer will see the issue and have time to correct it before the application reaches the hands of the user. However, compile time analysis is limited to those errors which can be determined by a careful analysis of a region of source code. The Solaris Studio Code Analyzer User Guide has a list of the types of errors that can be identified through static analysis.An example of the kind of error that can be identified through static analysis is the use of uninitialized data shown in the following code:#include <stdio.h>void main(){ int i; printf("i = %i\n", i);}The reporting of these kinds of static errors is a two step process. The first step is to compile the application with the flag -xprevise that enables static analysis. The second step is to use one of the two tools that produce a report on the detected errors. The tool codean produces a textual report, the code-analyzer tool presents the data in a GUI. The following output shows compiling the code and seeing the textual report.$ cc -g -xprevise uni.c$ codean a.outSTATIC report of a.out:ERROR 1 (UMR): accessing uninitialized data: i at: main() <uni.c : 6> 3: void main() 4: { 5: int i; 6:=> printf("i = %i\n", i); 7: }PREVISE SUMMARY for a.out: 1 error(s), 0 warning(s), 0 leak(s) in totalThe following figure shows the same report in the code-analyzer GUI:In the GUI, the type of error is shown as the title, the line containing the error is highlighted in grey.Although static analysis is relatively limited in what it can achieve, it will report errors on all the previously shown code snippets. Consider the earlier code containing an access to an uninitialized variable.ERROR 1 (UMR): accessing uninitialized data: i at: main() <uninit.c : 11> 8: void main() 9: { 10: int i; 11:=> printint(&i); 12: }PREVISE SUMMARY for a.out: 1 error(s), 0 warning(s), 0 leak(s) in totalNote that the error is reported at the point where the variable is passed into the printint() routine. This is not a speculative error, the tool has actually looked into printint() and identified that the variable i is read there, and so passing in an uninitialized variable is an error. If the code is changed so that the variable is initialized in the routine, then the compiler no longer reports the function call as a problem. It is able to do this kind of detailed analysis because it can see the bodies of both the calling and called functions, and is therefore able to see both how the variable is initialized and how it is later used.For the code where there is an access to previously freed memory, the tool reports both the location of the access and the location where the memory was freed. This greatly helps the developer find the program flow where there is a problem, and makes it much easy to resolve the issue.WARNING 1 (FMR): reading from freed memory: i at: main() <freed.c : 13> 10: { 11: int * i = malloc(sizeof(int)); 12: free(i); 13:=> printint(i); 14: } was freed at: main() <freed.c : 12> 9: void main() 10: { 11: int * i = malloc(sizeof(int)); 12:=> free(i); 13: printint(i);However, this is not the only issue that the tool picks up in this code. It also identifies a missing check for malloc() returning a null pointer. This missing return error is quite common in code, and is relatively easy for the compiler to identify. It is worth noting that in the event of an error the program should check the errno variable to determine whether whether the error condition is fatal or indicates an action that should be retried. In the case of malloc() it is possible for the OS to report that it currently has insufficient memory available, but the program should retry the allocation operation in the expectation that more memory will have been made available.WARNING 2 (MRC): missing null-pointer check after malloc: malloc(4) at: main() <freed.c : 11> 7: } 9: void main() 10: { 11:=> int * i = malloc(sizeof(int)); 12: free(i);PREVISE SUMMARY for ./a.out: 0 error(s), 2 warning(s), 0 leak(s) in totalThe earlier example which shows an access past array bounds is too complex for the tool to diagnose because in this example the bounds checking would require the tool to iterate through the loop, so detecting this error would require dynamic checking of the code. However, the tool does pick up two other issues with the code. Once again the code contains no check of the return from malloc(). The second issue is that the tool detects a memory leak where the allocated memory is not freed.LEAK 1 : 1 block left allocated on heap with a total size of 4 bytes main() <array.c : 14> 10: } 12: void main() 13: { 14:=> int * array = (int*) calloc(sizeof(int), 10); 15: printarray(array,11);WARNING 1 (MRC): missing null-pointer check after malloc: calloc(4,10) at: main() 10: } 12: void main() 13: { 14:=> int * array = (int*) calloc(sizeof(int), 10); 15: printarray(array,11);PREVISE SUMMARY for ./a.out: 0 error(s), 1 warning(s), 1 leak(s) in totalIt is impossible for static analysis of applications to identify all the errors that exist in an application. The risk is that more extensive static analysis of the code may end up raising many false positives. However, there are a surprising number of real errors that can be identified. The big advantage of using this kind of detection is that it all happens at compile time long before any users have had the chance to try the code.

There is a subset of memory access errors that can be detected at compile time. This is obviously the ideal time to warn about these issues because the developer will see the issue and have time...

Oh, no! What Have I Done Now? - Common Types of Memory Access Errors

Darryl Gove and I wrote a paper on memory access errors and how we are now detecting them in real time. Not forgetting the next step, we also provide great tools to solve these errors. Darryl has moved on to other things, so it is up to me convert the paper into a series of blogs. Why blogs? They seem to be much easier to find and are good at providing morsels of information.I have left many of Darryl's spelling mistakes in this particular blog - that's just my way of acknowledging him.Memory access errors represent one of the most pervasive, hard to detect, and often the most destructive types of programmer error. These happen when an application accesses memory that has not been initialised, is no longer valid, or is outside the bounds of the allowed area.A classic example of this is the “Heartbleed” security flaw where a determined hacker could extract information from the internals of an application by requesting that the application return data past the end of an internal buffer.There are essentially three major kinds of memory access errors.Uninitialised memoryConsider the following code:#include <stdio.h> void printint(int *i) { printf("Integer = %i\n",*i); } void main() { int i; printint(&i); }The problem code declares a variable, and then later in the program flow reads from the memory location where the variable resides. It’s important to note that the declaration and read are arbitrarily far apart, potentially in both the source code and in runtime. In this example the variable resides on the stack, but it could also be held in memory allocated dynamically using malloc().Accessing freed memoryConsider the following code:#include <stdio.h>#include <stdlib.h>void printint(int *i){ printf("Integer = %i\n", *i);}void main(){ int * i = malloc(sizeof(int)); free(i); printint(i);}In this example we are allocating a buffer, freeing it, then reading from it. This is an interesting kind of error because the memory location may well still be valid and may even contain the correct value, but on some occasions the buffer may have been reused by the time the invalid access takes place. If the buffer has already been reused, then reading from it will not return the expected result, and writing to it would cause data corruption. Of course, the time interval between the free() and the invalid access could be arbitrarily long, so it could be very hard to determine the actual cause of the error.Accessing past the end of allocated memoryA very common kind of error is where the access is past the end of the allocated region. For example:#include <stdio.h>#include <stdlib.h>void printarray(int *array, int len){ for (int i=0; i<len; i++) { printf("Index %i = %i\n", i, array[i]); }}void main(){ int * array = (int*) calloc(sizeof(int), 10); printarray(array,11);}In the example, the code allocates a chunk of memory of sufficient size to hold ten integers. This memory is passed into a function that prints out the elements in the array, but for demonstration purposes the code attempts to access the 11th element of the array.Other detectable memory errorsBeyond these three broad categories of memory error there are a large number of other detectable kinds of error. It is useful to take a look at the list of errors detectable by the tool discover which is available as part of the Studio.These errors are problems like double freeing memory, or passing a corrupt address to free, or even overlapping addresses to memcpy() (which expects non-overlapping addresses unlike memmove()).

Darryl Gove and I wrote a paper on memory access errors and how we are now detecting them in real time. Not forgetting the next step, we also provide great tools to solve these errors. Darryl has...

Tired of Purify? Valgrind Too Slow and Bloated? Use Discover

Long ago Purify was my memory access checker of choice. I do have fond memories of Purify finding some of my nastiest coding mistakes. I loved it, but slowly it started to look old and tired.Now I use Studio Discover to find my mistakes before my customers do. To make it an easy transition from Purify to Discover, I have created a table that maps common Purify errors to Discover errors. Most of them are quite intuitive, so you should have no problem in transitioning. Purify Description Discover Description ABR Array Bounds Read ABR beyond Array Bound Read ABW Array Bounds Write ABW beyond Array Bound Write BSR Beyond Stack Read SBR beyond Stack frame Bounds Read BSW Beyond Stack Write SBW beyond Stack frame Bounds Write FMM Freeing Mismatched Memory BFM Bad Free Memory FMR Free Memory Read FMR Freed Memory Read FMW Free Memory Write FMW Freed Memory Write FNH Freeing Non Heap Memory BFM Bad Free Memory FUM Freeing Unallocated Memory BFM / DFM Bad Free Memory / Double Free Memory IPR Invalid Pointer Read IMR Invalid Memory Read IPW Invalid Pointer Write IMW Invalid Memory Write MIU Memory In Use ALLOC Use Discover Function API MLK Memory Leak LEAK Memory Leak MSE Memory Segment Error IMR / IMW Invalid Memory Read / Write PLK Potential Memory Leak LEAK Warning Potential Memory Leak SBR Stack array Bounds Read ABR beyond Array Bound Read SBW Stack array Bounds Write ABW beyond Array Bound Write UMC Uninitialized Memory Copy no need We suppress UMR for paddings UMR Uninitialized Memory Read UMR Uninitialized Memory Read ZPR Zero Page Read IMR Invalid Memory Read ZPW Zero Page Write IMW Invalid Memory Write Purify does not have these errors that Discover reports, but these are interesting enough to mention here FRP Freed Realloc Parameter OLP Overlapping source and destination PIR Partially Intialized Read AZS Allocating Zero Size Memory

Long ago Purify was my memory access checker of choice. I do have fond memories of Purify finding some of my nastiest coding mistakes. I loved it, but slowly it started to look old and tired.Now I use...

Dear Raj, I Have Thousands of Code Analysis Errors. What do I do?
-- Distraught in Software City

Dear Distraught in Software City,Take a deep breath! You will feel better when I tell you that this is a common ailment. I have just the right prescription for you.When you run a code analysis tool - static, dynamic, coverage - for the very first time, it is typical to see a huge number of errors. This is discouraging and an all too common response is to put it all away for another day when you will have more time. And, of course, that day never comes.Studio Code Analyzer has a feature that can help you cope with this. Run the tools regularly and use codean what-is-new feature to know of any new problems that got introduced. You can fix the newly introduced problems and attack the older problems slowly one by one, or en masse if time permits. We use it everyday and compare against the results we saved the previous day. It is much easier to be on top of these as they are usually few and far between.The example below is for Previse (static analyzer), but same works for Discover and Uncover. % cc -xprevise t.c % codean --save a.out % vi t.c int *q = malloc(100*sizeof(int)); 22: add_0_1_put_in_2(p);PREVISE SUMMARY: 0 new error(s), 0 new warning(s), 1 new leak(s) in total % codean --save a.out % vi t.c int *q = malloc(100*sizeof(int)); 21: free(q);PREVISE SUMMARY: 0 fixed error(s), 0 fixed warning(s), 1 fixed leak(s) in total

Dear Distraught in Software City, Take a deep breath! You will feel better when I tell you that this is a common ailment. I have just the right prescription for you.When you run a code analysis tool -...

Move Over Purify and Valgrind,
There is a New Kid in Town - Real Time Memory Access Checker

If you have used any software memory access checkers (if you haven't, you should), such as, Purify and Valgrind, you know they are a bit slow. We are talking 40x, 60x, even more than that over the baseline timing of your program. Our own Discover which is considerablyfaster could be impractical to use on something that normally runs, say, for one whole day. Admittedly, this is going to be a bit of a tease, but how about a real time memoryaccess checker. It could change how we manage memory access errors today. The usual scenario is that I get a a bug, I have a test case that shows my program behaving weirdly. I suspect I have overflown a buffer (gone beyond bounds of an array, for those who don't care for the latest lingo, ahem! that's me) or I mismanaged memory by using a free'd memory area. Twenty years ago, I used to whip out my trusty Purify and, voila!, there was a good chance that I would spot the problem. Today I do thesame, only with Studio Discover.So, yes, I figure out the problem little faster with Discover, however itis essentially the same process.We promise to give you more details in a few days, but what if we coulddo this checking in real time? It is definitely going to make bug investigationmuch faster and feasible in many more cases. A bigger change couldcome from being able to activate the checker during deployment. If youhave been using standard C/C++ memory management routines, you wouldnot even have to modify your program to do this. Not even rebuild. You can use the same binary. Here is the kind of difference I am seeing with ADI for the geomean slowdown of a bunch of programs I tried. If you want to try it yourself before you have an ADI enabled system in your closet, sign up on swisdev.oracle.com.

If you have used any software memory access checkers (if you haven't, you should), such as, Purify and Valgrind, you know they are abit slow. We are talking 40x, 60x, even more than that over...

Finding Memory Leaks in a Server or a Long Running Program

Here is a tip. You may know that you can call Studio Code Analyzer Discover functions from your program to get memory leak and memory allocation information. For example, you can call discover_report_unreported_leaks() to see the leaks that Discover knows about at any given point in your program. The complete list of functions is here.If you have a long running program or a server that never exits, you can call these functions using the debugger, dbx, at any time even if you haven't put the calls in your code. The program, of course, must be Discover'ed earlier at least in the lite mode, option -l. Note that dbx can attach to a running program.Below is an example. a.out is a long running program with two processes with one leak each. I wrote a script, rl, that has the commands to ask the program to report unreported leaks. You give it a process id. You can use the script again and again to see new leaks, if any. % cat rl #!/bin/sh dbx - $1 > /dev/null 2>&1 0) { 16: 17:=> void *p = malloc(64); 18: printf("%jd: Parent allocation 64\n", (intmax_t)getpid()); 19: p = 0; 20: for (int j=0; j < 1000; j++) sleep(1); _start() + 0x66 **********************************************************Now for the child process % rl 8253The program reports ******** discover_report_unreported_leaks() Report ******** 1 allocation at 1 location left on the heap with a total size of 32 bytes LEAK 1: 1 allocation with total size of 32 bytes main() + 0x80 21: } 22: 23: else { 24:=> void *p = malloc(32); 25: printf("%jd: Child allocation 32\n", (intmax_t)getpid()); 26: p = 0; 27: for (int j=0; j < 1000; j++) sleep(1); _start() + 0x66 **********************************************************

Here is a tip. You may know that you can call Studio Code Analyzer Discoverfunctions from your program to get memory leak and memory allocation information. For example, you can...

Oracle

Integrated Cloud Applications & Platform Services