Linux uses the memory for disk cache. When data is loaded from disk, Linux keeps it in memory in case it needs the data again. Since this allocated memory is not actively in use, it can be cleared whenever another process needs memory. However, if nothing else requests memory, the disk cache continues to occupy that memory, and Linux considers the memory to be used rather than free. This looks like a memory leak, even though it is not. (Nagios in particular exhibits this issue.)
Example from a Linux VM:
total used free shared buffers cached
Mem: 2123456 1704528 418928 27116 100800 924500
-/+ buffers/cache: 679228 1444228
Swap: 2170876 0 2170876
- Total memory (green) = ~2G
- Used memory (red) = ~1.7G
- Buffers and cached = ~1.1G
- Swap is not used, so we don’t worry about it.
- -/+ buffers/cache
- The first number is used memory minus buffers + cache. The second number is free memory plus buffers+cache.
- Total - used = free (2123456 – 1704528 = 418928)
- Real used = used - buffers - cached (1704528 – 100800 – 924500 = 679228)
- Real free = free + buffers + cached (418928 + 100800 + 924500 =1444228)
- Because buffers and cache are available for other processes, the actual amount of memory used is 679228, and free memory is 1444228.
|It looks like 80% of the memory is being used, but in reality Linux only consumes 32%.|
Example from a FireEye appliance
Output from the FireEye CLI:
cms-4310-1 # sh memory
Total Used Free Used+B/C Free-B/C
Physical 32239 MB 2912 MB 29327 MB 12437 MB 19801 MB
Swap 16384 MB 255 MB 16129 MB
Physical Memory Borrowed for System Buffers and Cache:
Buffers: 303 MB
Cache: 9221 MB
Total Buffers/Cache: 9525 MB
Total 32239MB, used 2912MB free 29327MB.
Output from the Linux command shows a different result:
[czw@cms-44310-1 ~] # free -m
total used free shared buffers cached
Mem: 32239 12414 19824 0 303 9211
-/+ buffers/cache: 2898 29340
Swap: 16384 254 16129
"Used" is significantly larger in Linux than in the FireEye CLI because
The Linux free command does not do this.
The SNMP Side
FireEye does not have any specific OID for memory; we use a standard MIB (HOST-RESOURCES). This MIB works same way on a Linux box, taking the numbers from free output. Here is how it looks like in snmpwalk:
The first block is the description:
HOST-RESOURCES-MIB::hrStorageDescr.1 = STRING: Physical memory
HOST-RESOURCES-MIB::hrStorageDescr.3 = STRING: Virtual memory
HOST-RESOURCES-MIB::hrStorageDescr.6 = STRING: Memory buffers
HOST-RESOURCES-MIB::hrStorageDescr.7 = STRING: Cached memory
HOST-RESOURCES-MIB::hrStorageDescr.8 = STRING: Shared memory
HOST-RESOURCES-MIB::hrStorageDescr.10 = STRING: Swap space
The next block is the size of the memory:
HOST-RESOURCES-MIB::hrStorageSize.1 = INTEGER: 33012776
HOST-RESOURCES-MIB::hrStorageSize.3 = INTEGER: 49790216
HOST-RESOURCES-MIB::hrStorageSize.6 = INTEGER: 33012776
HOST-RESOURCES-MIB::hrStorageSize.7 = INTEGER: 9433620
HOST-RESOURCES-MIB::hrStorageSize.8 = INTEGER: 0
HOST-RESOURCES-MIB::hrStorageSize.10 = INTEGER: 16777440
From this example, you can see that Physical memory is 33012776, Virtual memory (= physical + swap) is 49790216, max memory buffers is same as max physical memory, and swap size is 16777440.
Check the used memory sizes:
HOST-RESOURCES-MIB::hrStorageUsed.1 = INTEGER: 12715724
HOST-RESOURCES-MIB::hrStorageUsed.3 = INTEGER: 12976632
HOST-RESOURCES-MIB::hrStorageUsed.6 = INTEGER: 311528
HOST-RESOURCES-MIB::hrStorageUsed.7 = INTEGER: 9433620
HOST-RESOURCES-MIB::hrStorageUsed.10 = INTEGER: 260908
- Used virtual memory (.1) = 12715724, although we expected around 2912MB from the show memory output.
- Memory buffers (.6) is 311528 and cached memory (.7) is 9433620. If we use the logic above, the real physical memory usage is 12715724 – 311528 – 9433620 = 2970576, or 2900MB (which is shown in the CLI).
To get the real number, we have to query .1, .6 and .7 and calculate properly. The OIDs are: