HugePages for Sybase / SAP ASE databases on linux

[jgleichmann]

Hello SYB DBAs,

you may be noticed a message in your DB log which looks like this:

"Could not allocate memory using Huge Pages. Allocated using regular pages. For better performance, reboot the server after configuring enough Huge Pages."

Don't worry about it, if no Huge Pages are available, the system automatically switch back to the normal page size of the OS. For example in Linux the regular memory page size is 4KB.

The huge page size you can find out via command:

cat /proc/meminfo | grep Hugepagesize

Hugepagesize:       2048 kB

Hugepages speed up the memory management.

Hugepages allocated in one piece and can't be paged out into the paging space.

So you can ensure that the DB memory never be paged out.

Please read also this notes about HugePages and Sybase/SAP ASE:

2021541 - SAP ASE runs slow due to table scans with regular page allocation

1805750 - SYB: Usage of huge pages on Linux Systems with Sybase ASE

For example if the sybase instance need 10GB without Hugepages:

10*1024*1024/4kb=2621440  (4kb Pages)

Every Page needs a 8byte entry in the page table 2621440 * 8byte = 20971520 => 20MB

For example if the sybase instance need 10GB with Hugepages:

10*1024*1024/2048kb=5120 (2mb Pages)

Every Page needs a 8byte entry in the page table 5120 * 8byte = 40960 => 40KB

This values need to be multiplicated with the user processes, but the shared memory must be considered!

If you need more information about the Page Table, TLB and Huge Pages you should have a look at stefan.koehler blog which is regarding oracle but however the facts are the same:

http://scn.sap.com/community/oracle/blog/2013/06/17/oracle-myths-and-common-misconceptions-about-tra...

If you want to use HugePages follow this steps as root

1) calculate the needed memory

select cast(sc.name as varchar(30)) as name, (scc.value*2/1024) as MB

from sysconfigures sc, syscurconfigs scc

where sc.name like 'total%memory' and sc.config = scc.config

go

name                           MB

------------------------------ -----------

total logical memory                  6028

total physical memory                 6400

=> Round the ASE "max memory" (total physical memory) setting up to the next closest 256MB, divide by the Hugepagesize, and configure at least that many huge pages. "

=> 6400 / 256mb = 25 => OK

2) calculate the needed HugePages

When ASE uses huge pages, it will allocate memory to the nearest multiple of 256Mb so don’t configure the ASE exactly to the maximum number of huge pages, but leave a small amount unused (e.g. 300/400 Mb). In case a small increase in size is needed due to the change of a Sybase config you will not get into trouble.

6400+400=6800

6800*1024/2048=3400

vi /etc/sysctl.conf

vm.nr_hugepages=3400

3) activate the kernel settings, if enough memory is available

sysctl -p

if the memory is not free, restart the server or add memory

4) Allow Sybase ASE owner to make use of available HugePages

vi /etc/security/limits.conf

<Sybase ASE OS owner>   soft memlock unlimited

<Sybase ASE OS owner>   hard memlock unlimited

5) check the config

cat /proc/meminfo | grep Huge

HugePages_Total:    3400

HugePages_Free:        0

HugePages_Rsvd:        0

HugePages_Surp:        0

Hugepagesize:       2048 kB

6) start the Sybase/SAP ASE and you will see this message in the logs

"Allocated memory using Huge pages."

7) after DB start

cat /proc/meminfo | grep Huge

HugePages_Total:    3250

HugePages_Free:     1655

HugePages_Rsvd:     1605

HugePages_Surp:        0

Hugepagesize:       2048 kB

If you get the same error as before extend the HugePages.

more offical info:

1805750 - SYB: Usage of huge pages on Linux Systems with Sybase ASE

Before and after process details

Here are the before and after effects regarding the dataserver (sybase DB process under unix) after a fresh restart:

Before:

pmap <sid>

START               SIZE     RSS     PSS   DIRTY    SWAP PERM MAPPING

0000000000400000  32736K  15460K  15460K      0K      0K r-xp /sybase/SMP/ASE-16_0/bin/dataserver

00000000025f7000   5692K   2664K   2664K    216K      0K rwxp /sybase/SMP/ASE-16_0/bin/dataserver

0000000002b86000   2488K   1196K   1196K   1196K      0K rwxp [heap]

0000000142df4000 6291456K 1507196K 1507196K 1507172K      0K rwxs /SYSVba156435

00007fffed33e000     20K     16K      2K      0K      0K r-xp /lib64/libnss_dns-2.11.3.so

[...]

Total:           6507456K 1530568K 1528772K 1509564K      0K

ps auxw | head -1; ps auxw | grep <sid>

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

sybsmp   31464 15.5  9.2 6507456 1516852 ?     Ssl  08:56   0:24 /sybase/SMP/ASE-16_0/bin/dataserver -sSMP -d/sybase/SMP/sybsystem/master.dat -e/sybase/SMP/ASE-16_0/install/SMP.log -c/sybase/SMP/ASE-16_0/SMP.cfg -M/sybase/SMP/ASE-16_0 -i/sybase/SMP -N/sybase/SMP/ASE-16_0/sysam/SMP.properties

=> RSS = 1,5GB

=> VSZ = 6,5GB

After:

pmap <sid>

START               SIZE     RSS     PSS   DIRTY    SWAP PERM MAPPING

0000000000400000  32736K  16096K  16096K      0K      0K r-xp /sybase/SMP/ASE-16_0/bin/dataserver

00000000025f7000   5692K   2664K   2664K    216K      0K rwxp /sybase/SMP/ASE-16_0/bin/dataserver

0000000002b86000   2648K   1244K   1244K   1244K      0K rwxp [heap]

00002aaaaac00000 6291456K      0K      0K      0K      0K rwxs /SYSVba156435

00007fffed33e000     20K     16K      0K      0K      0K r-xp /lib64/libnss_dns-2.11.3.so

00007fffed343000   2044K      0K      0K      0K      0K ---p /lib64/libnss_dns-2.11.3.so

00007fffed542000      4K      4K      4K      4K      0K r-xp /lib64/libnss_dns-2.11.3.so

00007fffed543000      4K      4K      4K      4K      0K rwxp /lib64/libnss_dns-2.11.3.so

00007fffed544000     48K     20K      0K      0K      0K r-xp /lib64/libnss_files-2.11.3.so

00007fffed550000   2044K      0K      0K      0K      0K ---p /lib64/libnss_files-2.11.3.so

[...]

Total:           6507616K  24056K  22094K   2440K      0K

vmsaplnx02:~ # ps auxw | head -1 ; ps auxw | grep <sid>

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

sybsmp    1716  9.6  0.1 281536 22712 ?        Ssl  09:12   0:20 /sybase/SMP/ASE-16_0/bin/dataserver -sSMP -d/sybase/SMP/sybsystem/master.dat -e/sybase/SMP/ASE-16_0/install/SMP.log -c/sybase/SMP/ASE-16_0/SMP.cfg -M/sybase/SMP/ASE-16_0 -i/sybase/SMP -N/sybase/SMP/ASE-16_0/sysam/SMP.properties

=> RSS = 23MB

=> VSZ = 282MB

If you compare the RSS/VSZ of the 2 ouput you will see the benefit.

Hope this howto helps you to use your memory ressources optimal.

Thanks for reading and sharing!

Please feel free to ask or get in contact directly if you need if you need assistance.

Best Regards,

Jens Gleichmann

Technology Consultant at Q-Partners Consulting und Management GmbH (www.qpcm.eu)

References

Virtual memory - Wikipedia, the free encyclopedia

Resident set size - Wikipedia, the free encyclopedia

[Oracle] Myths and common misconceptions about (transparent) huge pages for Oracle databases (on Lin...

2021541 - SAP ASE runs slow due to table scans with regular page allocation

1805750 - SYB: Usage of huge pages on Linux Systems with Sybase ASE