Information Guide:  Linux and Memory as disk cache

Version 7

    DOC-6352

     

    Linux uses the memory for disk cache. When data is loaded from disk, Linux keeps it in memory in case it needs the data again. Since this allocated memory is not actively in use, it can be cleared whenever another process needs memory. However, if nothing else requests memory, the disk cache continues to occupy that memory, and Linux considers the memory to be used rather than free. This looks like a memory leak, even though it is not. (Nagios in particular exhibits this issue.)

     

    Example from a Linux VM:

    root@czw-debian:~# free

                total       used       free     shared     buffers     cached

    Mem:      2123456    1704528     418928      27116      100800     924500

    -/+ buffers/cache:    679228    1444228

    Swap:     2170876          0    2170876

    • Memory
      • Total memory (green) = ~2G
      • Used memory (red) = ~1.7G
      • Buffers and cached = ~1.1G
      • Swap is not used, so we don’t worry about it.
    • -/+ buffers/cache
      • The first number is used memory minus buffers + cache. The second number is free memory plus buffers+cache.

    The Math:

    • Total - used = free (21234561704528 = 418928)
    • Real used = used - buffers - cached (1704528100800924500 = 679228)
    • Real free = free + buffers + cached  (418928 + 100800 + 924500 =1444228)
    • Because buffers and cache are available for other processes, the actual amount of memory used is 679228, and free memory is 1444228.

     

    It looks like 80% of the memory is being used, but in reality Linux only consumes 32%.

     

     

    Example from a FireEye appliance

     

    Output from the FireEye CLI:

    cms-4310-1 # sh memory

             Total      Used      Free     Used+B/C  Free-B/C

    Physical 32239 MB   2912 MB  29327 MB  12437 MB  19801 MB

    Swap     16384 MB    255 MB  16129 MB

     

    Physical Memory Borrowed for System Buffers and Cache:

      Buffers:                         303 MB

      Cache:                        9221 MB

      Total Buffers/Cache:  9525 MB

     

    Total 32239MB, used 2912MB free 29327MB.

     

    Output from the Linux command shows a different result:

    [czw@cms-44310-1 ~] # free -m

                total       used      free    shared    buffers    cached

    Mem:        32239      12414      19824        0        303      9211

    -/+ buffers/cache:      2898      29340

    Swap:       16384        254      16129

     

    "Used" is significantly larger in Linux than in the FireEye CLI because
    the CLI has already deducted buffers and cached from the used memory.

     

    The Linux free command does not do this.

     

     

    The SNMP Side

     

    FireEye does not have any specific OID for memory; we use a standard MIB (HOST-RESOURCES). This MIB works same way on a Linux box, taking the numbers from free output. Here is how it looks like in snmpwalk:

     

    The first block is the description:

    HOST-RESOURCES-MIB::hrStorageDescr.1 = STRING: Physical memory

    HOST-RESOURCES-MIB::hrStorageDescr.3 = STRING: Virtual memory

    HOST-RESOURCES-MIB::hrStorageDescr.6 = STRING: Memory buffers

    HOST-RESOURCES-MIB::hrStorageDescr.7 = STRING: Cached memory

    HOST-RESOURCES-MIB::hrStorageDescr.8 = STRING: Shared memory

    HOST-RESOURCES-MIB::hrStorageDescr.10 = STRING: Swap space

    The next block is the size of the memory:

    HOST-RESOURCES-MIB::hrStorageSize.1 = INTEGER: 33012776

    HOST-RESOURCES-MIB::hrStorageSize.3 = INTEGER: 49790216

    HOST-RESOURCES-MIB::hrStorageSize.6 = INTEGER: 33012776

    HOST-RESOURCES-MIB::hrStorageSize.7 = INTEGER: 9433620

    HOST-RESOURCES-MIB::hrStorageSize.8 = INTEGER: 0

    HOST-RESOURCES-MIB::hrStorageSize.10 = INTEGER: 16777440

    From this example, you can see that Physical memory is 33012776, Virtual memory (= physical + swap) is 49790216, max memory buffers is same as max physical memory, and swap size is 16777440.

     

    Check the used memory sizes:

    HOST-RESOURCES-MIB::hrStorageUsed.1 = INTEGER: 12715724

    HOST-RESOURCES-MIB::hrStorageUsed.3 = INTEGER: 12976632

    HOST-RESOURCES-MIB::hrStorageUsed.6 = INTEGER: 311528

    HOST-RESOURCES-MIB::hrStorageUsed.7 = INTEGER: 9433620

    HOST-RESOURCES-MIB::hrStorageUsed.10 = INTEGER: 260908

    • Used virtual memory (.1) = 12715724, although we expected around 2912MB from the show memory output.
    • Memory buffers (.6) is 311528 and cached memory (.7) is 9433620. If we use the logic above, the real physical memory usage is 12715724 – 311528 – 9433620 = 2970576, or  2900MB (which is shown in the CLI).

     

    The Math:

    To get the real number, we have to query .1, .6 and .7 and calculate properly. The OIDs are:

    .1.3.6.1.2.1.25.2.3.1.6.1

    .1.3.6.1.2.1.25.2.3.1.6.6

    .1.3.6.1.2.1.25.2.3.1.6.7

     

     

    Thanks to Support Engineer kszalai for contributing this article and to amitkarpe and george.anderson for keeping it up to date!