XtraBackup causing swap despite numa_interleave on mysqld

Jeremy Cole has an excellent article on why a mysqld instance will swap despite proper memory settings on modern hardware here. A quick hand-wave of a summary goes: in modern architecture, every process is, by default, limited to a 1/Nth of the available memory on a machine, where N is the number of cores, and will start to swap on disk when all the available memory in that 1/Nth is consumed.

 

The eventual solution for a large, single-instance mysqld-running machine is to enable numa_interleave in the mysqld_safe section of the config:

Screen Shot 2014-11-18 at 10.09.03 AM

Implicitly, when that flag is set, mysqld is being started by mysqld_safe with a numactl flag that will spread the memory allocation evenly across all nodes (“numactl –interleave=all [… rest of mysqld start command]”).

 

There is a subtlety, though, that must be considered: mysqld was just started up and allocated most of the memory on each of the nodes. When all nodes on a machine are mostly full, no locally running script, such as XtraBackup, can consume significant memory without starting to cause swap since that script, by default, will only consume memory in its hosting node. Also, that swap, generally, will be the mysqld process.

 

The solution, then, becomes to interleave all processes that lead to swap or any process that can reasonably be expected to consume a significant amount of memory before it leads to swap.

 

Before, Xtrabackup could look like this:

innobackupex –defaults-extra-file=/path/to/auth –tmpdir=/backup/partition –sla[…]

Now, Xtrabackup will look like this:

numactl –interleave=all innobackupex –defaults-extra-file=/path/to/auth –tmpdi[…]

 

I hope this helps resolve any additional swap issues even after mysqld’s numa_interleave flag was enabled. Let me know if there are clarifications I can make.

 

Update Aug 2015: I’ve learned about a few more variables related to numa management. Will provide corrections soon.

Leave a Reply