Many a times you raise an incident with SAP Support about a performance issue on Java server, only to realize that you missed saving the necessary logs. The problem you face now is : You have already restarted the system or the problem seems to disappear but you still need the root cause lest the problem should appear again
This article provides you some guidelines on which logs need to be collected before you trigger that all important "restart" of Java server.
1) Slow Java start-up : That moment when you wait forever for the state to change from starting application/framework to running
- Thread Dumps(Most Important) : When the start-up or the system even when it is up hangs, trigger thread dumps. To do so, use these notes depending on the Netweaver release
1955835 - AS Java server node hangs at starting phase - What To Do for 7.0X
1953845 - AS Java server node hangs at starting phase - What To Do for 7.10+
You need to capture at-least 3-4 dumps at an interval of 1 minute. This is done to get a complete picture of the hang situation. For example, if a thread consistently shown as blocked/hanged in all the thread dumps then the problem lies there. You can also do thread dump analysis on your own using the tool in SAP note#1020246 - Thread Dump Viewer for SAP Java Engine
Apart from the above method, you can use SAPJVM profiler tool as well. The SAP JVM Profiler helps to analyze the resource consumption of a Java application running on a SAP Java VM.
To collect profiling information for a slow start-up , use the SAP Knowledge Base Article(KBA) 1995883 - Analyzing slow AS-JAVA startup using the SAPJVM profiler. Send the output *.prf file to SAP Support.
For additional methods of taking thread dumps, click here
2) High CPU consumption : You have situations where some Java processes start consuming High CPU. It is very important to collect the right logs while the issue is happening. Thread dumps come handy here as well but for systems running on SAPJVM, it is recommended to use SAPJVM profiler to collect the required data. For this, use the SAP KBA#1783031 - Analyzing AS Java performance with SAP JVM Profiler
3) Outofmemory issues. Many a times you'll see Java server restarts/becomes unresponsive with "outofmemory"(dev_serverX log in work directory says "JControlCloseProgram: good bye... (exitcode = 666)" or "java.lang.OutOfMemoryError "). The most important log file is the heap dump generated during the crash. This dump will usually be found /usr/sap/<SID>/<instance>/j2ee/cluster/serverX. It will normally have the format *.hprof. The size of this file will be equal to the heap memory consumption at the time of the crash. Send this file to SAP support before deleting it permanently. This file is automatically generated if the profile parameter is set. Note 1004225 - How to create a full HPROF heap dump of J2EE engine ; will help you in setting these parameters.
4) Slow response from the server/ enterprise portal: Here again thread dumps and SAPJVM profiler output are needed. Capture the logs when the issue is occurring in the same way as for slow startup.
6) It is always recommended to attach these logs while raising an incident 1) Logs mentioned in 1867207 - Collecting traces to troubleshoot Netweaver AS Java runtime/startup issues 2) SAPMMC snapshot : Refer to KBA 1847251 How to create an MMC snapshot about an SAP system. If the issue is not reproducible at will, mention the last time-stamp when the issue occurred(of course, attach the logs from the same time period). In addition, do inform about any changes that were done on the system.
Of course for all the above cases,the all important default trace, work directory(in most cases) and system information are required too.
- For relevant default traces refer to 1596214
- Work directory is located at /usr/sap/<SID>/<instance>/work
- For system information refer to 1757810 or 1752501
