System debugging and analysis techniques for ASE on Unix/Linux - Part 6

[Former Member]

Blog #6 – Memory usage

    A. When to use these tools

Most of ASE’s memory usage will be in shared memory. However, there are times when you may want to  examine what other memory ASE may be using. For instance, if your system is showing signs of memory contention you may want to verify exactly what memory ASE is using, and what it is using it for. Note also that these tools can be used to examine other processes that may be running on the system to take a look at their memory usage as well.

    B. What to look for

An important concept to understand is that on Unix/Linux a process will have two values for memory usage. There is virtual memory, which is (roughly) the amount of memory the process has told the OS that it might use; and there is resident set memory, which is the actual physical memory in use by the process. The key concept is that virtual memory has no direct relationship to how much memory a process is using. To illustrate, here are a couple examples taken from a Linux system using the ‘ps’ command. The VSZ column is the virtual memory size and the RSS column is the resident set size. On linux these sizes are given in KB.

USER PID %CPU %MEM    VSZ   RSS TTY STAT START   TIME COMMAND

root 2441  0.0  0.0 535296 5200 ?        Ssl  Jul08 0:08 automount --pid-file /var/run/autofs.pid

dputz 21350  9.4  1.0 618624 156588 ?       Ssl 11:27   0:01 /work/dputz/ase157/ASE-15_0/bin/dataserver

Here we see that the automount system daemon has over 500MB of virtual memory but is only using about 5MB of real memory. The dataserver process has just a little more virtual memory (600MB) but is using 156MB of real memory.

When we are looking at memory usage we will want to concentrate on the real memory in use by a process – that is what is actually taking up space in the system’s memory.

    C. What tools to use

There are several useful tools we can use to look at memory usage. In the section above we saw how the ‘ps’ command can show the resident set size on Linux. In addition, there are utilities such as pmap, procmap, ipcs, and top that can let us look at processes memory usage. These can vary quite a bit by platform; here is a short list of what is available and what they show:

Linux

------

• ps aux - Shows VSZ (virtual memory size) and RSS (resident set size) for all processes on the

                    system.

• pmap -x <process id> - Shows the Kbytes (virtual memory size) and RSS (resident set size) for

                                         each address space in the processes memory map. Note that this will

                                 show memory usage for all shared libraries separately as well. The ASE

                                 shared memory will show up as well – see the example 3. below.

• ipcs -ma - Shows the shared memory segments in use on the system. If the “nattch” column    

                       shows a 0 that indicates that there are no running  programs currently attached.

• top -a – Shows the top users of memory, sorted in descending order by resident set size (RES).

• cat /proc/meminfo – Can be used to help show if ASE is using huge pages.

Solaris

--------

• ps -elfy  - Shows SZ (virtual memory size) and RSS (resident set size) for all processes on the

                     system.

• pmap -x <process id> - Shows the Kbytes (virtual memory size) and RSS (resident set size) for

                                         each address space in the processes memory map. Note that this will

                                         show memory usage for all shared libraries separately as well. The ASE

                                         shared memory will show up as well.

• ipcs -ma - Shows the shared memory segments in use on the system. If the “NATTCH” column                

                        shows a 0 that indicates that there are no running programs currently attached.

• top -o res – Shows the top users of memory, sorted in descending order by resident set size (RES).

AIX

--- 

• ps axv  - Shows SIZE (virtual memory size for just the data segment of the process) and RSS

                     (resident set size) for all processes on the system.

• procmap <process id> - Shows the virtual memory size for each address space in the

                      processes memory map. Note that this will show memory usage for all shared libraries

                      separately as well. Note that on AIX the ASE shared memory will not show up in this list.

• ipcs -ma - Shows the shared memory segments in use on the system. If the “NATTCH” column

                       shows a 0 that indicates that there are no running programs currently attached.

• topas -P – Shows an output similar to the ps command. It will include the DATA RES and TEXT

                      RES columns, which make up the processes resident set size. This is an interactive

                     display, you can pick which column to sort on by using the left and right arrow keys.

• svmon -P <process id> - Shows a list of all memory segments and sizes for the specified process.

                      It will also show memory usage by pagesize, which can help identify if ASE is using any

                      large pages.

HP-UX

------ 

• ps -elf  - Shows SZ (resident set size) for all processes on the system. Note that on HP-UX you can

                    set the environment variable UNIX_STD=2003 and get both resident set size (SZ) an

                    virtual memory size (VSZ) by running ps with a command line like

                   ps -A -o pid -o pcpu -o sz -o vsz -o comm

• pmap <process id> - Shows the VSZ(virtual memory size) and RSZ (resident set size) for

                      each address space in the processes memory map. Note that this will

                     show memory usage for all shared libraries separately as well. The ASE

                     shared memory will show up as well at the end of this output.

• ipcs -ma - Shows the shared memory segments in use on the system. If the “NATTCH” column               

                       shows a 0 that indicates that there are no running programs currently attached.

• top -  Shows the top users of cpu, sorted in descending order by current cpu usage. Virtual

                 memory size (SIZE) and resident set size(RES) are shown.

• glance – This is an HP-UX interactive utility. To see processes sorted in order by the resident set

                     size; on the main screen enter “o” followed by “1”. Skip down to the “sort key” and

                     type “rss” , enter, then “y”.

    D. Examples

1. A question arises on a Linux system showing memory paging regarding exactly how much memory ASE is using. The ASE “max memory” is set to 3720000 (about 7.5GB)

and “allocate max shared memory” is set to 0.

1. ps output

USER       PID %CPU %MEM    VSZ RSS TTY      STAT START   TIME COMMAND

sybase    21293 0.6  5.9 8382844 906676 ?      Ssl 11:25   0:57 /work2/ase16/ASE-16_0/bin/dataserver

1. pmap output

Address           Kbytes     RSS Dirty Mode   Mapping

0000000000400000   32720 12492       0 r-x--  dataserver

00000000025f3000    5684 2048     172 rwx--  dataserver

0000000002b80000    1488 208     208 rwx--    [ anon ]

0000000004b4a000    1352 1220    1220 rwx--    [ anon ]

0000000144c6c000 7341860  885920  885920 rwxs-    [ shmid=0xe8012 ]

0000003735e00000     128 116       0 r-x--  ld-2.12.so

000000373601f000       4 4       4 r-x-- ld-2.12.so

0000003736020000       4 4       4 rwx--  ld-2.12.so

…

----------------  ------ ------  ------

total kB         8382844  906676 888784

Note that on Linux, “Dirty” pages are those that the process has written to; meaning that if the system needed to reuse that memory it would forst have to write those pages out to swap space.

From these two outputs we can see that, while the virtual memory size of ASE is quite large (over 8GB) the actual resident set memory being used is less than 1GB.

     2. On AIX, the system administrator has collected ‘ps’ output that shows the SZ column

for each of 8 ASE engines to be much less than the configured ‘max memory’ of about 2GB. They want to know how much memory ASE is really using.

1. The issue here is that AIX does not include shared memory in the resident set size
2. The ps avxw output shows:

      PID TTY STAT  TIME PGIN  SIZE RSS   LIM  TSIZ TRS %CPU %MEM COMMAND

  8060938 pts/5 A     0:00    0 10768 33292    xx 44555 22524  0.0 0.0 /sybase/ase157/ASE-15_0/bin/dataserver -ONLINE:7,0

  8192142 pts/5 A     0:01    0 10764 33288    xx 44555 22524  0.0 0.0 /sybase/ase157/ASE-15_0/bin/dataserver -ONLINE:3,0

  8847466 pts/5 A     0:00   21 12096 34620    xx 44555 22524  0.0  0.0 /sybase/ase157/ASE-15_0/bin/dataserver -ONLINE:1,0

  7864604 pts/5 A     0:50 1203 13624 36148    xx 44555 22524  0.1 0.0 /sybase/ase157/ASE-15_0/bin/dataserver -sdwpaix157

  7930130 pts/5 A     0:00    2 10768 33292    xx 44555 22524  0.0 0.0 /sybase/ase157/ASE-15_0/bin/dataserver -ONLINE:2,0

  8847650 pts/5 A     0:00    0 10768 33292    xx 44555 22524  0.0 0.0 /sybase/ase157/ASE-15_0/bin/dataserver -ONLINE:6,0

  8913190 pts/5 A     0:00    0 10768 33292    xx 44555 22524  0.0 0.0 /sybase/ase157/ASE-15_0/bin/dataserver -ONLINE:5,0

  8978768 pts/5 A     0:01    9 11996 34520    xx 44555 22524  0.0 0.0 /sybase/ase157/ASE-15_0/bin/dataserver -ONLINE:4,0

The RSS columns (in KB units on AIX) all add up to only about 270 MB. In order to calculate the actual total memory in use we will need to add in the size of the

shared memory. To get that, we can use the “ipcs –ma” command. It shows:

T        ID KEY        MODE       OWNER GROUP  CREATOR   CGROUP NATTCH     SEGSZ CPID  LPID   ATIME DTIME    CTIME

m  78643205 0x970ef015 --rw-------    sybase ipsec    sybase    ipsec 8 2252800000 7864604 8060938 12:04:48 12:04:48 12:04:39

The SEGSZ gives us the size of the shared memory (in bytes) which we can see matches the configured “max memory”.

     c. Another way to look at this would be to use the “svmon” command; taking one of the dataserver process ids:

svmon -P 7930130

-------------------------------------------------------------------------------

     Pid Command          Inuse      Pin Pgsp  Virtual 64-bit Mthrd  16MB

7930130 dataserver      610150 558551        0   604497 Y     Y     N

     PageSize                Inuse        Pin       Pgsp Virtual

     s 4 KB              562662     550007          0 557009

     m 64 KB                2968        534          0       2968

     L 16 MB                   0          0          0          0

This shows 562662 4KB pages and 2968 64KB pages in use; which all added up comes to about 2.4GB.

       3. Someone has asked whether ASE running on Linux is experiencing a “memory leak”; which is a condition where memory gets allocated by a program but never freed. They have monitored the resident set size of ASE from boot time and see the following pattern:

At boot – resident set size = 1023000

After running 1 hour resident set size was 1392248

After running 2 hours it was 3101496

After running 3 hours it was 3588204

After running 4 hours it was 3925660

Then after running a large batch application they saw it grow to 7952832

After that, it stayed fairly constant, with little change for a couple of weeks.

They questioned whether this showed that ASE had some sort of memory leak as it booted up and ran for a while.

Using the pmap –x utility we were able to show that all of the growth occurred in the

Shared memory segment. On linux, that line in the pmap output looked like:

Address           Kbytes     RSS Dirty Mode   Mapping

0000000000400000   32720 15064       0 r-x--  dataserver

00000000025f3000    5684 2648     172 rwx--  dataserver

0000000002b80000    1488 208     208 rwx--    [ anon ]

0000000003df3000    1352 1220    1220 rwx--    [ anon ]

0000000143f15000 7341860 6977384 3977384 rwxs-    [ shmid=0x130012 ]

…

This showed that the resident set size growth was due entirely to ASE using more and more of the data cache spaces in the shared memory segment. There was no memory leak; this is normal and expected behavior as ASE uses more and more of the cache.

    E. Conclusion

Using the system tools to look at memory usage can help determine what the complete memory usage of ASE (and other processes) is. This will help in diagnosing any issues that might arise from a system being low on memory (such as memory paging). i