Lei Zhilong

The best way to input is to output

Nov 12, 2018 - 4 minute read - Comments - Linux

Case Study: Swapoff Cannot Allocate Memory

Description

On some of our CentOS Linux nodes, we use Zabbix to monitor host resources. Sometimes, Zabbix reports system swap space high usage when the used size is larger then 50% of its total space, which indicates there may not be sufficient RAM. After cleaning up some unused tasks, available memory reaches up to 90GB+ according to the result of free -h command while the total swap space is only 10GB. However, when I try to clean up the swap space by using swapoff /dev/dm-1 && swapon /dev/dm-1, I still get this error:

1

swapoff: /dev/dm-1: swapoff failed: Cannot allocate memory

It seems that swap space cannot be switched off when it is in use even though enough RAM is provided.

Analyze

When I looked this issue up online, some said swapoff fails because the amount of swap file being used is bigger than what can fit in the RAM. Clearly, sufficient RAM does not guarantee that swapoff can be performed successfully.

Also, there are some answers to this issue saying that enable memory over-committing may be able to solve this.

1

sysctl -w vm.overcommit_memory=1

Well, that didn’t work for my situation either.

There are several notions that are only loosely related.

  • Physical memory usage
  • Swap area usage
  • Virtual memory reservation
  • Virtual memory allocation

When a process allocated memory (malloc or similar), what it does is actually reserving memory. By default, Linux over-commits virtual memory so doesn’t care that much about reservation and generally return successfully.

When you access reserved memory for the first time, this memory has to be backed by pages located either on RAM or on disk.

If there is not enough RAM to hold all the accessed virtual memory on RAM, the system starts to paginate (what is usually called swapping) and the performance stalls.

If processes reserve more memory than the sum of the swap area and (part of the) RAM, and your system is configured not to over-commit memory, the allocation fails. This can happen even while you have plenty of RAM free and are not using any pages on the swap area. This is the price to pay to have a system that doesn’t randomly kills applications.

In that case, even thought I have enough free virtual memory but a part of it is reserved so demand some swap area to be present which means I cannot remove the latter.

I guess one likely practical solution would be create another swap space and use it to exchange the old one temporarily.

Solution

  1. Check swap status

    1
2
3

    swapon -s
Filename                                Type                 Size         Used     Priority
/dev/dm-1                              partition            3145724       1524700    -1

  2. Create a file using dd as a temporary swap space

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

    dd if=/dev/zero of=/home/swap bs=1024 count=512000

512000+0 records in
512000+0 records out
524288000 bytes (524 MB) copied, 1.2637 s, 415 MB/s

mkswap /home/swap
mkswap: /home/swap: warning: don't erase bootbits sectors
        on whole disk. Use -f to force.
Setting up swapspace version 1, size = 511996 KiB
no label, UUID=155bb4be-29cb-4e95-9581-24e11be6188f

  3. swapon the temporary swap file and check out free memory.

    1
2
3
4
5
6
7
8

    # this would take a few minutes.
swapon /home/swap

free -m
                    total       used       free     shared    buffers     cached
Mem:                 3832       3550       282          0          4        166
-/+ buffers/cache:   3378       453
Swap:                3571       1495       2076

  4. swapoff the swap device which should be cleaned up and swapon it again after that

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

    
# a few more minutes depending on you swap size
swapoff /dev/dm-1

free -m
                    total       used       free     shared    buffers     cached
Mem:                 3832       3711        120          0          2         19
-/+ buffers/cache:   3689       143
Swap:                499        499          0

swapon /dev/dm-1
free -m
                    total       used       free     shared    buffers     cached
Mem:                3832       3717        114          0          1        25
-/+ buffers/cache:  3691       140
Swap:               3571       495         3076

  5. swapoff the temporary swap file and remove it from you disk.

    1
2
3
4
5
6
7
8

    swapoff /home/swap
free -m
                    total       used       free     shared    buffers     cached
Mem:                3832       3725        106          0          0              15
-/+ buffers/cache:  3710       122
Swap:               3071       476         2595

rm -f /home/swap

Reference