When examining free memory in Linux, it seems that the reported total free memory is a static value that tells you how much memory is physically installed in a system.

However, interestingly, this is not the case.

For instance, on my 64 GiB x86-64 system:

$ grep MemTotal /proc/meminfo
MemTotal:       65113764 kB

Hmm, that’s ~62.1 GiB, not 64 GiB?

First of all, let’s confirm I really do have 64 GiB installed. x86 systems provides metadata about hardware via its Desktop Management Interface (DMI).

This is exposed in /sys/firmware/dmi/, but it’s more easily obtained via the dmidecode tool:

$ sudo dmidecode -t memory
Getting SMBIOS data from sysfs.
SMBIOS 3.7.0 present.

Handle 0x0015, DMI type 17, 100 bytes
Memory Device
	...
	Size: 16 GiB
	...
Handle 0x0018, DMI type 17, 100 bytes
Memory Device
	...
	Size: 16 GiB
	...
Handle 0x001C, DMI type 17, 100 bytes
Memory Device
	...
	Size: 16 GiB
	...
Handle 0x001F, DMI type 17, 100 bytes
Memory Device
	...
	Size: 16 GiB
	...

So that confirms it – I have 64 GiB, or 67,108,864 KiB installed.

That leaves us with 67,108,864 – 65,113,764 = 1,995,100 KiB or ~1.9 GiB unaccounted for.

That’s a substantial chunk of memory! Where did it all go?

Early boot MemTotal

First things first – during boot a snapshot of memory state is output to the kernel ring buffer, we can locate it thusly:

$ dmesg | grep "Memory:"
[    0.273630] Memory: 64666024K/66413500K available (21873K kernel code, 3061K rwdata, 16828K rodata, 4820K init, 5240K bss, 1627252K reserved, 0K cma-reserved)

This only begs further questions!

Firstly – this tells us that we have only 66,413,500 KiB memory available, rather than the 67,108,864 KiB we know is physically installed.

That leaves us with a 695,364 KiB or ~679 MiB gap. We will come back to this.

Secondly – how are we to interpret the rest of what’s reported?

We see that we have 64,666,024 KiB (~61.6 GiB) memory free at this point in the boot – but that’s not very useful information.

What’s more important is – what is the MemTotal at this stage?

Confusingly, the reported categories of used memory (kernel code, rwdata, etc.) are not memory currently allocated, but rather named portions of the reported ‘reserved’ memory.

What is reserved memory? Well, unfortunately this term is rather overused in Linux. In this context, it means memory that is unavailable to the physical allocator – i.e. exactly what we are looking for – memory subtracted from MemTotal.

So the MemTotal at this point in time is available physical memory less reserved memory.

This is 66,413,500 KiB – 1,627,252 KiB = 64,786,248 KiB (~61.8 GiB).

But when the system is booted, we have a MemTotal of 65,113,764 KiB! That leaves us with a 327,516 KiB (~320 MiB) difference.

The reason for this is that some of this reserved memory is freed back to the kernel during boot.

We can observe this from the kernel ring buffer:

$ dmesg | grep "Freeing"
[    0.130587] Freeing SMP alternatives memory: 56K
[    0.278484] efi: Freeing EFI boot services memory: 194448K
[    0.429105] Freeing initrd memory: 123964K
[    1.709961] Freeing unused decrypted memory: 2028K
[    1.710375] Freeing unused kernel image (initmem) memory: 4820K
[    1.710588] Freeing unused kernel image (text/rodata gap) memory: 652K
[    1.710755] Freeing unused kernel image (rodata/data gap) memory: 1604K

(Note that the ‘SMP alternatives’ memory is in fact not strictly freed back to us)

That gives us 194,448 + 123,964 + 2,028 + 4,820 + 652 + 1,604 = 327,516 KiB which is exactly the difference between these numbers.

That means we have 1,627,252 – 327,516 = 1,299,736 = ~1.24 GiB reserved at runtime.

Summing the kernel portion of this from above gives us 21,873 + 3,061 + 16,828 + 4,820 + 5,240 – 4,820 – 652 – 1,604 = 44,746 KiB = ~43.7 MiB for the kernel image.

Subtracting this from the known reserved memory gives us 1,299,736 – 44,746 KiB = 1,254,990 KiB = ~ 1.2 GiB of other reserved memory.

So accounting for all we know:

  • 65,113,764 KiB = ~62.1 GiB MemTotal
  • 1,254,990 KiB = ~1.2 GiB other reserved memory
  • 695,364 KiB = ~679 MiB unaccounted memory
  • 44,746 KiB = ~43.7 MiB kernel image

The memory map

By far the bulk of the remaining reserved memory is accounted for by ‘the memory map’, which is how the kernel tracks physical memory.

For an x86-64 system, the base page size is 4 KiB, and for each of these pages the kernel maintains a struct page metadata object to do this.

We can use the pahole utility (available in UEK and debian systems via the ‘dwarves’ package) to figure out the size of a struct page:

$ pahole -s page 2>/dev/null | awk '{print $2}'
64

And thus we can calculate the space used from this.

We track memory this way for everything that is not unaccounted for, which is 65,113,764 KiB + 44,746 KiB + 1,254,990 KiB = 66,413,500 KiB, in other words the total physically available memory from the early boot dmesg output.

At a base page size of 4 KiB this is 16,603,375 pages. With each struct page taking 64 bytes, this means that the memory map occupies ~1,037,711 KiB or ~ 1 GiB.

The rule of thumb here is – 1/64 (~1.6%) of the total installed ram.

This accounts for the bulk of the ‘other’ reserved memory, 1,254,990 – 1,037,711 = 217,279 KiB or ~212 MiB, leaving us with:

  • 65,113,764 KiB = ~62.1 GiB MemTotal
  • 1,037,711 KiB = ~1 GiB memory map
  • 695,364 KiB = ~679 MiB unaccounted memory
  • 217,279 KiB = ~212 MiB other reserved memory
  • 44,746 KiB = ~43.7 MiB kernel image

Note that the ongoing folio and memdesc work by Oracle engineer Matthew Wilcox aims to reduce the memory map significantly which should result in a larger MemTotal in future kernel releases.

It’s worth noting here that HugeTLB, and specifically its HugeTLB Vmemmap Optimization (HVO) feature, can reduce the amount of memory used in the memory map (configuration option CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP).

Kernel early reserved memory

Let’s go hunting for that 212 MiB. To do that, we’re going to have a dig a little deeper to get the information we need.

Luckily, everything that takes meaningful chunks of this announces itself in the kernel ring buffer.

You can dig through manually, but let’s start with the biggest portion – data used for internal hash tables which ends up reserved:

$ dmesg | grep 'hash table entries'
[    0.042030] Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes, linear)
[    0.042982] Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes, linear)
[    0.130943] Mount-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.130982] Mountpoint-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.274037] posixtimers hash table entries: 65536 (order: 8, 1048576 bytes, linear)
[    0.274037] futex hash table entries: 32768 (2097152 bytes on 1 NUMA nodes, total 2048 KiB, linear).
[    0.359127] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.365281] IP idents hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.372788] tcp_listen_portaddr_hash hash table entries: 32768 (order: 7, 524288 bytes, linear)
[    0.372827] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.372990] TCP established hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[    0.373230] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, linear)
[    0.373439] MPTCP token hash table entries: 65536 (order: 9, 1572864 bytes, linear)
[    0.373566] UDP hash table entries: 32768 (order: 9, 2097152 bytes, linear)
[    0.373731] UDP-Lite hash table entries: 32768 (order: 9, 2097152 bytes, linear)

We can grab the total hash data (dentry, inode, etc.) used via:

$ dmesg | grep 'hash table entries' | grep -Eo '[0-9]+ bytes' | awk '{print $1}' | paste -sd+ | bc
120852480

Giving us 118,020 KiB or ~116 MiB used for hash tables.

After this software I/O TLB metadata, percpu and node data take up the bulk of what remains:

$ dmesg | grep -E "software IO TLB: mapped|percpu: Embedded|NODE_DATA"
[    0.004126] NODE_DATA(0) allocated [mem 0x103d595280-0x103d5bffff]
[    0.039707] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u262144
[    0.387727] software IO TLB: mapped [mem 0x00000000b9800000-0x00000000bd800000] (64MB)

There’s 128 logical CPUs on the system and at 63 4 KiB pages a core this gives us 32,256 KiB or ~31.5 MiB reserved for per-CPU initial data.

Node data (NUMA system metadata of 0x103d5bffff – 0x103d595280 + 1 bytes or ~171 KiB.

In total we have:

  • 118,020 KiB = ~116 MiB hash tables (dentry, inode et al.)
  • 65,536 KiB = 64 MiB software I/O TLB
  • 32,256 KiB = 31.5 MiB per-CPU initial data
  • 171 KiB node metadata

This gives us 215,928 KiB or ~211 MiB in total. Accounting for alignment and any other small accounting data that the kernel may store that fully describes all of the accounted memory we’ve observed.

Summarising:

  • 65,113,764 KiB = ~62.1 GiB MemTotal
  • 1,037,711 KiB = ~1 GiB memory map
  • 695,364 KiB = ~679 MiB unaccounted memory
  • 118,020 KiB = ~116 MiB kernel hash tables (dentry, inode et al.)
  • 65,536 KiB = 64 MiB software I/O TLB
  • 44,746 KiB = ~43.7 MiB kernel image
  • 32,256 KiB = 31.5 MiB per-CPU initial data
  • 1,467 KiB = ~1.4 MiB other kernel housekeeping data

Firmware stolen memory

The 679 MiB of memory that remains unaccounted for is memory that wasn’t even available for reservation in the early stages of the boot process – it has been ‘stolen’ by the firmware.

All memory in the system is located at ‘physical’ addresses, these are addresses that tell the hardware exactly where the memory is located (i.e. which DIMM and where in the DIMM).

We can figure out the physical memory ranges that our RAM is installed at using the dmidecode tool:

[~]$ sudo dmidecode -t 19
# dmidecode 3.7
Getting SMBIOS data from sysfs.
SMBIOS 3.7.0 present.

Handle 0x0012, DMI type 19, 31 bytes
Memory Array Mapped Address
	Starting Address: 0x00000000000
	Ending Address: 0x000BFFFFFFF
	Range Size: 3 GiB
	Physical Array Handle: 0x0011
	Partition Width: 4

Handle 0x0013, DMI type 19, 31 bytes
Memory Array Mapped Address
	Starting Address: 0x00100000000
	Ending Address: 0x0103FFFFFFF
	Range Size: 61 GiB
	Physical Array Handle: 0x0011
	Partition Width: 0

So the RAM on my system is installed at physical addresses 0 – 0x000bfffffff and 0x00100000000 – 0x0103fffffff inclusive (summing to 64 GiB exactly as expected).

Some devices can be accessed via physical addresses, which sit outside of these ranges. These do not ‘steal’ from installed physical memory.

However some do sit within these ranges, so we must distinguish between the two.

Linux exposes memory used by system RAM and firmware in /sys/firmware/memmap/ which contains numbered entries for each range of memory exposed to the kernel.

So, looking at non-system RAM ranges that sit within the known physical RAM we get (with some bash trickery to get sorted ranges):

$ ls /sys/firmware/memmap/0/
end  start  type
$ # Lower ranges
$ for d in /sys/firmware/memmap/*; do start=$(<$d/start); end=$(<$d/end); type=$(<$d/type); if [[ $type != "System RAM" && $start -le 0x000bfffffff && $end -ge 0 ]]; then echo "$(($start)) $start $end $type = $((($end - $start + 1)/1024)) KiB"; fi; done | sort -n | awk '{$1=""}1'
 0xa0000 0xfffff Reserved = 384 KiB
 0x4000000 0x4039fff ACPI Non-volatile Storage = 232 KiB
 0x99ff000 0x9ffffff Reserved = 6148 KiB
 0xb000000 0xb020fff Reserved = 132 KiB
 0x8ed05000 0x94d04fff Reserved = 98304 KiB
 0x94d05000 0x9587efff ACPI Tables = 11752 KiB
 0x9587f000 0x9787efff ACPI Non-volatile Storage = 32768 KiB
 0x9787f000 0x9d9fefff Reserved = 99840 KiB
 0x9f757000 0x9f75bfff ACPI Non-volatile Storage = 20 KiB
 0x9fffb000 0xa7ffffff Reserved = 131092 KiB
 0xaa700000 0xb7ffffff Reserved = 222208 KiB
 0xbd800000 0xbfffffff Reserved = 40960 KiB
$ # Upper ranges
for d in /sys/firmware/memmap/*; do start=$(<$d/start); end=$(<$d/end); type=$(<$d/type); if [[ $type != "System RAM" && $start -le 0x0103fffffff && $end -ge 0x00100000000 ]]; then echo "$(($start)) $start $end $type = $((($end - $start + 1)/1024)) KiB"; fi; done | sort -n | awk '{$1=""}1'
 0xff000000 0x10000ffff Reserved = 16448 KiB
 0x103c000000 0x103c7fffff Reserved = 8192 KiB
 0x103d5c0000 0x103fffffff Reserved = 43264 KiB

Note however that part of the 0xff000000 – 0x10000ffff range is outside of the System RAM, so we subtract 16,384 from the total and arrive at 695,360 KiB which is within one 4 KiB page of our missing range, of which 44,772 KiB (~43.7 MiB) is ACPI data, and the remaining 650,592 KiB is stolen for other firmware purposes.

We’ve now identified exactly where the memory is stolen from, and matched our sought-after value. That leaves us with:

  • 65,113,764 KiB = ~62.1 GiB MemTotal
  • 1,037,711 KiB = ~1 GiB memory map
  • 650,592 KiB = ~635.3 MiB other memory used by the firmware
  • 118,020 KiB = ~116 MiB kernel hash tables (dentry, inode et al.)
  • 65,536 KiB = 64 MiB software I/O TLB
  • 44,772 KiB = ~43.7 MiB ACPI data
  • 44,746 KiB = ~43.7 MiB kernel image
  • 32,256 KiB = 31.5 MiB per-CPU initial data
  • 1,467 KiB = ~1.4 MiB other kernel housekeeping data

This is about as much information as you can get from the kernel!

Firmware ranges vary depending on hardware configuration, and much of it is based on non-public or difficult to obtain hardware and BIOS-specific information, however this memory usage is critical to the systems operation.

To summarise – MemTotal is equal to the installed memory, less early reserved kernel memory and memory stolen by the firmware.

Small note for the really keen: You can check /sys/firmware/efi/runtime-map/ to figure out how much memory EFI needs to keep in use (on my system ~94 MiB).