/contrib/famzah

Enthusiasm never stops


2 Comments

MemAvailable metric for Linux kernels before 3.14 in /proc/meminfo

A great new metric has been introduced in “/proc/meminfo” in the Linux 3.14 kernel — MemAvailable:

An estimate of how much memory is available for starting new applications, without swapping. Calculated from MemFree, SReclaimable, the size of the file LRU lists, and the low watermarks in each zone.

The estimate takes into account that the system needs some page cache to function well, and that not all reclaimable slab will be reclaimable, due to items being in use. The impact of those factors will vary from system to system.

I recommend that you read the kernel commit description for further details.

Since many people are still using Linux kernels before 3.14, I’ve backported this kernel patch to Perl. You can download the sources from GitHub: https://github.com/famzah/linux-memavailable-procfs

Many system administrators rely on the “free” tool to get a quick overview of the system’s memory usage. Unfortunately, the latest “procps” package still doesn’t interpret the “MemAvailable” metric, and even if it did, we don’t have it in Linux kernels before 3.14. Actually, the developers of the “procps-ng” package (which is the Debian, Fedora and openSUSE fork of “procps“) have reacted and did the same thing as me. For kernels before 3.14 they emulate the metric in the same way, and for kernels after 3.14, they display the native metric from “/proc/meminfo”. This makes my Perl port more or less redundant.

This is the reason I wrote a quick replacement of the “free” tool in Perl. A few examples of it follow.

Typical memory usage overview, in MBytes:

famzah@vbox:~$ ./free.pl -m
             total       used       free  anonymous     kernel     caches     others
Mem:          2488       1228       1259        608         24        580         15
  -/+ caches              648       1840
Swap:         1565          0       1565

Typical memory usage overview, in percentage:

famzah@vbox:~$ ./free.pl -mp
             total       used       free  anonymous     kernel     caches     others
Mem:          2488        49%        51%        24%         1%        23%         1%
  -/+ caches              26%        74%
Swap:         1565         0%       100%

Extended memory usage overview, in MBytes:

famzah@vbox:~$ ./free.pl -me
             total       used       free  anonymous     kernel     caches     others
Mem:          2488       1228       1260        608         24        580         14
  -/+ caches              647       1840
Swap:         1565          0       1565

Extended memory usage info:
  Buffers                  83
  Cached                  785
  SwapCached                0
  Shmem                   308
  AnonPages               300
  Mapped                  122
  Unevict+Mlocked           5
  Dirty+Writeback           0
  NFS+Bounce                0

Extended memory usage overview, in percentage:

famzah@vbox:~$ ./free.pl -mep
             total       used       free  anonymous     kernel     caches     others
Mem:          2488        49%        51%        24%         1%        23%         1%
  -/+ caches              26%        74%
Swap:         1565         0%       100%

Extended memory usage info:
  Buffers                  3%
  Cached                  32%
  SwapCached               0%
  Shmem                   12%
  AnonPages               12%
  Mapped                   5%
  Unevict+Mlocked          0%
  Dirty+Writeback          0%
  NFS+Bounce               0%

Memory logo

Advertisements


9 Comments

Know your Linux memory usage

I’ve given another try to understand the Linux memory usage statistics which are exported by “/proc/meminfo“. The main reason was to know how much memory is still available for free allocation by user applications, and also to understand the memory usage of the different Linux subsystems like the kernel, file system cache, etc.

The whole analysis is done on 649 machines with different workload. Most of them are running a 64-bit Linux kernel 3.2.59, while the others are running a 64-bit kernel with versions between 3.2.42 and 3.2.60. This is an LTS Linux kernel.

Cached memory and tmpfs

One of the most astonishing facts that I found out was that the “tmpfs” in-memory file system is counted against the “Cached” value in “/proc/meminfo”. The “tmpfs” file system is commonly mounted in “/dev/shm”. There is a separate stats key “Shmem” which shows the amount of memory allocated to “tmpfs” files, but at the same time this memory is added to the value of “Cached”. This is rather confusing because we’re used to think that both “Cached” and “Buffers” are reclaimable memory which can be freed anytime (or very soon enough) for use by applications or the kernel. That’s the current behavior of the “free” memory statistics tool which is widely used by System administrators to get an overview of the memory usage on their Linux systems. The “free” tool is part of the “procps” package which as of today is still not fixed and showing the “Cached” memory as free.

Here is a little proof:

famzah@vbox:~$ free -m
             total       used       free     shared    buffers     cached
Mem:          2488        965       1523          0         83        517
-/+ buffers/cache:        363       2124
Swap:         1565          0       1565

famzah@vbox:~$ dd if=/dev/zero of=/dev/shm/bigfile bs=1M count=800
838860800 bytes (839 MB) copied, 0.53357 s, 1.6 GB/s

famzah@vbox:~$ free -m
             total       used       free     shared    buffers     cached
Mem:          2488       1765        723          0         83       1317
-/+ buffers/cache:        364       2124
Swap:         1565          0       1565

The “cached” memory got 800 MB bigger. The free “-/+ buffers/cache” memory is still the same amount (2124 MB), which is wrong because the used memory in “tmpfs” cannot be reclaimed nor given to user applications or the kernel.

The reason that “tmpfs” is added to the “cached” memory usage is because “tmpfs” is implemented in the page cache which is accounted in the “cached” stats.

Active, Inactive, and Slab

The following is true for all machines that I tested on:

  • Active = Active(anon) + Active(file)
  • Inactive = Inactive(anon) + Inactive(file)
  • Slab = SReclaimable + SUnreclaim

Total memory usage

I tried to sum up the usage of the different fields in “/proc/meminfo” up to the maximum amount of available memory on each machine. The following two formulas both represent the whole memory usage:

  • MemTotal = MemFree + (Buffers + Cached + SwapCached) + AnonPages + (Slab + PageTables + KernelStack)
  • MemTotal = MemFree + (Active + Inactive) + (Slab + PageTables + KernelStack)

Both those formulas can be expanded to include the sub-components as well:

  • MemTotal = MemFree + (Buffers + Cached + SwapCached) + AnonPages + ((SReclaimable + SUnreclaim) + PageTables + KernelStack)
  • MemTotal = MemFree + ((“Active(anon)” + “Active(file)”) + (“Inactive(anon)” + “Inactive(file)”)) + ((SReclaimable + SUnreclaim) + PageTables + KernelStack)

The formulas give an error of up to -3% for only 6 machines. For the rest of the machines the error is less than +/- 1.5%. The less total available memory a machine has, the bigger the error. I suppose that this small error comes by the fact that not all “/proc/meminfo” counters are updated atomically at once.

Statistical keys

The following keys in “/proc/meminfo” seem to be accounted into other super-set keys, and are therefore only for statistical purpose:

  • Mapped
  • Dirty
  • Writeback
  • Unevictable
  • Shmem — accounted in “Cached”
  • VmallocUsed

References: