I’ve given another try to understand the Linux memory usage statistics which are exported by “/proc/meminfo“. The main reason was to know how much memory is still available for free allocation by user applications, and also to understand the memory usage of the different Linux subsystems like the kernel, file system cache, etc.
The whole analysis is done on 649 machines with different workload. Most of them are running a 64-bit Linux kernel 3.2.59, while the others are running a 64-bit kernel with versions between 3.2.42 and 3.2.60. This is an LTS Linux kernel.
Cached memory and tmpfs
One of the most astonishing facts that I found out was that the “tmpfs” in-memory file system is counted against the “Cached” value in “/proc/meminfo”. The “tmpfs” file system is commonly mounted in “/dev/shm”. There is a separate stats key “Shmem” which shows the amount of memory allocated to “tmpfs” files, but at the same time this memory is added to the value of “Cached”. This is rather confusing because we’re used to think that both “Cached” and “Buffers” are reclaimable memory which can be freed anytime (or very soon enough) for use by applications or the kernel. That’s the current behavior of the “free” memory statistics tool which is widely used by System administrators to get an overview of the memory usage on their Linux systems. The “free” tool is part of the “procps” package which as of today is still not fixed and showing the “Cached” memory as free.
Here is a little proof:
famzah@vbox:~$ free -m total used free shared buffers cached Mem: 2488 965 1523 0 83 517 -/+ buffers/cache: 363 2124 Swap: 1565 0 1565 famzah@vbox:~$ dd if=/dev/zero of=/dev/shm/bigfile bs=1M count=800 838860800 bytes (839 MB) copied, 0.53357 s, 1.6 GB/s famzah@vbox:~$ free -m total used free shared buffers cached Mem: 2488 1765 723 0 83 1317 -/+ buffers/cache: 364 2124 Swap: 1565 0 1565
The “cached” memory got 800 MB bigger. The free “-/+ buffers/cache” memory is still the same amount (2124 MB), which is wrong because the used memory in “tmpfs” cannot be reclaimed nor given to user applications or the kernel.
The reason that “tmpfs” is added to the “cached” memory usage is because “tmpfs” is implemented in the page cache which is accounted in the “cached” stats.
Active, Inactive, and Slab
The following is true for all machines that I tested on:
- Active = Active(anon) + Active(file)
- Inactive = Inactive(anon) + Inactive(file)
- Slab = SReclaimable + SUnreclaim
Total memory usage
I tried to sum up the usage of the different fields in “/proc/meminfo” up to the maximum amount of available memory on each machine. The following two formulas both represent the whole memory usage:
- MemTotal = MemFree + (Buffers + Cached + SwapCached) + AnonPages + (Slab + PageTables + KernelStack)
- MemTotal = MemFree + (Active + Inactive) + (Slab + PageTables + KernelStack)
Both those formulas can be expanded to include the sub-components as well:
- MemTotal = MemFree + (Buffers + Cached + SwapCached) + AnonPages + ((SReclaimable + SUnreclaim) + PageTables + KernelStack)
- MemTotal = MemFree + ((“Active(anon)” + “Active(file)”) + (“Inactive(anon)” + “Inactive(file)”)) + ((SReclaimable + SUnreclaim) + PageTables + KernelStack)
The formulas give an error of up to -3% for only 6 machines. For the rest of the machines the error is less than +/- 1.5%. The less total available memory a machine has, the bigger the error. I suppose that this small error comes by the fact that not all “/proc/meminfo” counters are updated atomically at once.
Statistical keys
The following keys in “/proc/meminfo” seem to be accounted into other super-set keys, and are therefore only for statistical purpose:
- Mapped
- Dirty
- Writeback
- Unevictable
- Shmem — accounted in “Cached”
- VmallocUsed
References:
- Linux kernel docs — the /proc file system
- Stackoverflow — unevictable page
- Linux kernel docs — unevictable LRU infrastructure
- stackexchange.com — Tracking down “missing” memory usage in linux
- linux-mm.org — Low On Memory
- Stackoverflow — Linux total available memory
- novell.com — Bug 405246 – Free memory reported by free, vmstat and top is wrong
- LINUX-VM mailing list — Amount of memory in buffers shown by the free command
- A fun image which represents the Linux Memory Management 🙂
May 9, 2017 at 1:31 pm
hi Ivan,
I’am trtacking a memory leak in home gateway. I have read a lot of posts about memory management in linux, but things still be confusing for me. For exampal how to get total free memory; Is it MemFree + (Buffers + Cached + SwapCached) + SReclaimable?
May 9, 2017 at 1:41 pm
I’am using the formula TotalFreeMemory = MemFree + (Buffers + Cached + SwapCached) + SReclaimable. TotalFreeMemory is decreasig a bout 9M in 3 days. I found that SUnreclaim has brown about 7.5 M. So there is 1.5 M lost. AnonPages has also grown about 1M.
What is the usage of AnonPages? is it normal that this bloc of memory grows like this?
Thanks
May 13, 2017 at 10:32 am
Take a look at the following project which is a working example on how to calculate the Total available (free) memory: https://github.com/famzah/linux-memavailable-procfs
If your Linux is using a modern “ps” and/or kernel, this info is already available there.
AFAIK, AnonPages is memory mapped by applications.
Please note that tracking free memory down to MBytes is error-prone, I think. I’m not sure if all counters which are exported are an atomic snapshot of the system.
May 15, 2017 at 2:16 pm
Thanks Ivan,
Could you please expalin to me how “user space” memory is conted? in fact, it seems to me that only kernel memory is counted in this formula : TotalFreeMemory = MemFree + (Buffers + Cached + SwapCached) + SReclaimable.
July 28, 2018 at 9:51 am
hi !
i have a question to disturb you.
i use ‘cat /proc/meminfo’ and got information of my centos6.5, Then i use the first formula in chapter of “Total memory usage”.
But there is too big difference between the sum of “MemFree + (Buffers + Cached + SwapCached) + AnonPages + (Slab + PageTables + KernelStack)” and MemTotal. WHY ??
July 28, 2018 at 1:37 pm
Hi, please provide the whole content of your “/proc/meminfo”.
July 30, 2018 at 5:27 am
[root@paas-online-devhttpproxy-001 ~]# uname -r
2.6.32-431.23.3.el6.x86_64
[root@paas-online-devhttpproxy-001 ~]# netstat -na|grep ESTABLISHED|wc -l
97580
[root@paas-online-devhttpproxy-001 ~]# cat /proc/meminfo
MemTotal: 3921112 kB
MemFree: 384496 kB
Buffers: 17440 kB
Cached: 161320 kB
SwapCached: 0 kB
Active: 2371380 kB
Inactive: 115236 kB
Active(anon): 2307884 kB
Inactive(anon): 328 kB
Active(file): 63496 kB
Inactive(file): 114908 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 1064 kB
Writeback: 0 kB
AnonPages: 2307860 kB
Mapped: 24608 kB
Shmem: 352 kB
Slab: 603656 kB
SReclaimable: 241756 kB
SUnreclaim: 361900 kB
KernelStack: 1952 kB
PageTables: 9316 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 1960556 kB
Committed_AS: 2983752 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 20388 kB
VmallocChunk: 34359715396 kB
HardwareCorrupted: 0 kB
AnonHugePages: 1982464 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 6144 kB
DirectMap2M: 4188160 kB
Result with your formula:
MemTotal(3486040) = MemFree(384496) + Buffers(17440) + Cached(161320) + SwapCached(0) + AnonPages(2307860) + Slab(603656) + PageTables(9316) + KernelStack(1952)
MemTotal(3422540) = MemFree(384496) + Active(2307884) + Inactive(115236) + Slab(603656) + PageTables(9316) + KernelStack(1952)
But the “Memtotal” in picture is 3921112 kb,there is too big difference.Why is this happening?
August 22, 2018 at 6:51 am
HI ,Ivan
i haven’t solve this problem,please help me ~~~
July 30, 2018 at 5:28 am
Although this core is 2.6.32,calculate with your formula,the inaccuracy of other server is acceptable.The phenomenon of this server is too strange.