When a program first reads or writes data to disk, the Linux operating system will also attempt to store the data in the page cache, which is allocated from unused portions of system memory.  The primary purpose of the page cache is to increase the performance of subsequent accesses to any data that happens to reside in page cache, instead of having to repeatedly read to/write from disk which is substantially slower.  However, if too much memory is being used by the page cache, this can potentially lead to long delays when a program later needs to allocate large amounts of contiguous memory.  In this scenario the operating system needs to clear the appropriate regions of page cache to make room for the new allocations, which may also involve having to write dirty pages to disk.   (Dirty pages are pages in the page cache that have been modified since the initial write to the page cache and disk - their data must be flushed to disk before freeing up their memory for reuse.) In the case where your machine has multiple NUMA nodes, if a particular NUMA node does not have sufficient free memory for an allocation because of high page cache usage, the machine may instead allocate memory on a different NUMA node that has sufficient free memory!  This will lead to substantially decreased memory read/write performance for your misplaced data.  (See my previous blog post here, for more on investigating this issue.) 

 

In the scenario where you suspect that the page cache is consuming a cumbersome amount of memory and wish to confirm it, there are some handy tools in Linux you can use.

 

The free command

A simple tool on Linux to look at memory usage including page cache is the free command.  The free command output might look something like this, on a machine with 64 gigabytes of memory: 

The number in the cached column on the far right indicates how much memory is used by the operating system for the page cache and slabs (learn more about slabs at http://www.secretmango.com/jimb/Whitepapers/slabs/slab.html).  However, the number does not break down the usage between page cache and slabs, nor does the free tool break down page cache usage between NUMA nodes.  

 

cat proc/memifo

For an even more detailed breakdown on memory usage, you can also try cat /proc/meminfo. An example output is shown below (output varies depending on your specific OS):

A lot of information is displayed here.  The MemTotal, MemFree, Buffers, and Cached fields are fairly self-explanatory and largely mirror the output of the free command.  In particular, the Cached field shows total memory used by the page cache.  Additionally, the Active(file) and Inactive(file) fields further categorize page cache usage: active page cache memory has been used recently and is (usually) not reclaimed until absolutely needed, whereas inactive page cache memory has not been used recently and therefore can be reclaimed without a large performance impact in general.

While using /proc/meminfo provides a very detailed view of memory usage, it does not break usage down per NUMA node.  If you wish to view information per node, then you may want to try…

 

numastat

A better tool for showing detailed memory usage per NUMA node, including page cache usage, is the numastat tool.  In order to show the detailed memory information, pass the –m argument to numastat upon execution:

 

In general, the fields displayed with numastat –m are the same as /proc/meminfo, except now the numbers are displayed per node (and curiously, the Cached field is not shown either.)  With this, you can now investigate if a particular NUMA node has higher page cache allocation than others.

 

Dropping the page cache

If you find yourself in a situation where too much memory is being consumed by the page cache, a last-resort option would be to flush the page cache entirely. Flushing the page cache frees up memory for other applications to use. Doing this requires some careful thought however, as clearing the cache can reduce system performance in other applications!  If you have decided that the benefits outweigh any potential consequences, then you can use echo 3 > /proc/sys/vm/drop_caches. This command clears the entire page cache. Note that this requires sudo privileges to execute.