More Linux Memory: Free memory that is not free, nor cache!

A customer contacted our support recently, wondering why his Linux servers showed high memory utilization.  We’ve talked a fair bit about monitoring Linux memory,  and what it means. But this case was a little different. The customer showed that according to free, top, and LogicMonitor graphs, most of his server’s memory was in use – and not as file system cache.

However, the odd thing is that top and ps showed that the sum of all processes RSS (resident segment size) was consuming very little memory. So where was the memory going?

The most common reason people have for complaining that the free memory shown in top does not add up to the resident memory used by processes is because they fail to notice that most of the memory is used by the cache (which is, in effect, free memory.)  But that was not the case here.

Looking at the contents of /proc/meminfo showed these two lines:

Slab: 4126212 kB
SReclaimable: 4096172 kB

So – almost 4G of memory was in use by the kernel slab memory structures – but almost all of that memory was reclaimable. (Or, in effect, free.)

So reclaimable slab space is yet another way that Linux memory can be in effect free, but not show up in the free memory statistics.

Does this change the way we should monitor the memory of Linux systems? Not at all. Linux will do a good job of managing the available physical memory and the data structures within it. When physical memory runs low, it will start swapping out to disk backed virtual memory. So long as you are monitoring the rate of blocks being swapped per second, and this is fairly low number – it doesn’t really matter what the physical memory is being used for.

Of course, you can dig into what is using the slab,  using slabtop, /proc/slabinfo, and start tuning your /proc/sys/vm stuff, for things like min_slab_ratio…

But, unless you have active paging on this system, I would not bother.
Everything is working fine, and if memory gets low, the kernel will give up the reclaimable slab (4G worth.)