SOFTWARE PROGRAM ABNORMALLY TERMINATED

[former_member219483]

Dear all,

i have a problem in one of our application server (dialog instance) when user run a Transaction the work process hanged and terminated .

our platform:

OS: AIX 7100-00-08-1241

Database: Oracle 11203

sap :7.40

we have 4 server for prd (two server db1 and db2 with hacmp and contain ci ,scs ,db and dilog) (two server for application app1 and app2 (addational dialog)

the errpt :

---------------------------------------------------------------------------

LABEL:          CORE_DUMP

IDENTIFIER:     A924A5FC

Date/Time:       Mon Sep  1 15:59:04 EET 2014

Sequence Number: 3243

Machine Id:      00F8895C4C00

Node Id:         ERPAPP1

Class:           S

Type:            PERM

WPAR:            Global

Resource Name:   SYSPROC

Description

SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes

SOFTWARE PROGRAM

User Causes

USER GENERATED SIGNAL

        Recommended Actions

        CORRECT THEN RETRY

Failure Causes

SOFTWARE PROGRAM

        Recommended Actions

        RERUN THE APPLICATION PROGRAM

        IF PROBLEM PERSISTS THEN DO THE FOLLOWING

        CONTACT APPROPRIATE SERVICE REPRESENTATIVE

Detail Data

SIGNAL NUMBER

          11

USER'S PROCESS ID:

              13500642

FILE SYSTEM SERIAL NUMBER

          16

INODE NUMBER

                  4148

CORE FILE NAME

/usr/sap/PRD/D00/work/core

PROGRAM NAME

disp+work

STACK EXECUTION DISABLED

           0

COME FROM ADDRESS REGISTER

getformat 48

kindly i need your help ASAP

Thanks&BR

Ahmed Ragab

[former_member219483 · Accepted solution]

Dear S Sriram,

sorry for delay respond because i moved the resource group appdata to the app1 server which we have the problem .

the problem happend  run a tcode FAGLB03 , the session hanged and i  observed the core file under (/usr/sap/PRD/D04) increase untill the fs /usr/sap reach to 100% and the work process terminated and the session close, i removed the core file but this not solve the problem .

the dump error :

ategory               Internal Kernel Error

untime Errors         SYSTEM_CORE_DUMPED

ate and Time          03.09.2014 10:06:51

Short Text

    Process terminated by signal 11.

What happened?

    Error in the SAP kernel

    The current ABAP program "SAPLFAGL_ITEMS_SELECT" had to be terminated because

     the ABAP

    processor detected a system error.

What can you do?

    Note which actions and entries caused the error to occur.

    Consult your SAP administrator.

    Using transaction ST22 for ABAP dump analysis, you can view, manage,

    and retain termination messages for longer periods.

Error analysis

    A process in the SAP system has been terminated by a signal in the

    operating system.

    Possible causes are:

    1) An internal error in the SAP system

    2) Process terminated externally (by the system administrator) with a

    signal.

    Last error logged in SAP kernel

    Component............ Taskhandler

    Location.......... SAP-Server ERPAPP_PRD_00 on host ERPAPP (wp 6)

    Version.............. 1

    Error code............ 16789511

    Error text............ ThSigHandler: Signal

    Description............

    System call..........

    Module.............. /bas/740_REL/src/krn/si/th/thxxhead.c

    Line................ 2341

    The error reported by the operating system is: Error number.....

    Error text....... " "

How to correct the error

    The SAP system work directory (/usr/sap/c11/D04/work for example) often

contains a file called 'core'.

Save this file under another name.

System environment

    SAP Release..... 740

    SAP Basis level 0002

    Application server... ERPAPP

    Network address...... 10.20.20.15

    Operating system... AIX

    Release.............. 7.1

    Hardware type....... 00F8895C4C00

    Character length..... 16 Bits

    Pointer length........ 64 Bits

    Work process number... 6

    Shortdump setting. full

    Database server... ERPDB1

    Database type..... ORACLE

    Database name..... PRD

    Database user ID SAPSR3

    Terminal.......... EGSHQITL011

    Character set C

    SAP kernel....... 740

    Created on....... Apr 10 2013 22:22:13

    Created at....... AIX 1 6 00C7B8024C00

    Database version OCI_112, 11.2.0.3.0, V1, default

    Patch level....... 12

    Patch text.......

Database............. , , ORACLE 11.2.*.*.*

SAP database version. 740

Operating system... , AIX 1 7

Memory consumption

Roll.... 0

EM...... 22849760

Heap.... 0

Page.... 106496

MM used. 4035088

MM free. 3655136

and Transaction

Client................. 300

User.................. ARAGAB

Language key.......... E

Transaction......... FAGLB03

Transaction ID...... 5406FA4ACC020620E10080000A14140B

EPP whole context ID...... 5406FBBBCC010D80E10080000A14140B

EPP connection ID........ 00000000000000000000000000000000

EPP call counter......... 0

Program.............. SAPLFAGL_ITEMS_SELECT

Screen.............. SAPMSSY0                                1000

Screen line.......... 6

Active debugger..... "none"

Information on where terminated

    The termination occurred in ABAP program "SAPLFAGL_ITEMS_SELECT", in

     "FAGL_GET_ITEMS_BSIS". The main program

    was "FAGL_ACCOUNT_ITEMS_GL                   ".

    In the source code, the termination point is in line 433 of (Include)

    program "LFAGL_ITEMS_SELECTU01".

.

[former_member219483 · Accepted solution]

dev_disp logs

***LOG Q0K=> DpMsAttach, mscon ( erpdb) [dpMessageSer 1513]

use SAPLOCALHOST=<ERPAPP> as internal hostname

MBUF state LOADING

DpStartStopMsg: send start message (myname is >ERPAPP_PRD_00                           <)

DpStartStopMsg: start msg sent to message server o.k.

MPI: dynamic quotas disabled.

MPI init: pipes=4000 buffers=6395 reserved=1918 quota=10%

Wed Sep  3 10:06:15 2014

*** WARNING => NiIHdlSetParamNoSslSessionDone: call on non SSL connection [nixxi.cpp    11068]

Wed Sep  3 10:06:15 2014

*** WARNING => NiIHdlSetParamNoSslSessionDone: call on non SSL connection [nixxi.cpp    11068]

Wed Sep  3 10:06:15 2014

WLM Tag 'PRD/DSP' successfully set for this process

Wed Sep  3 10:06:17 2014

CCMS: Initialized monitoring segment of size 60000000.

CCMS: Initialized CCMS Headers in the shared monitoring segment.

CCMS: Checking Downtime Configuration of Monitoring Segment.

CCMS: AlMsUpload called by wp 1024.

Wed Sep  3 10:06:18 2014

CCMS: AlMsUpload successful for /usr/sap/PRD/D00/log/ALMTTREE (534 MTEs).

Wed Sep  3 10:06:19 2014

CCMS: start to initalize 3.X shared alert area (first segment).

DpCheckPreemptionTicker: created thread for DpPremptionTicker

DpMBufHwIdSet: set Hardware-ID

***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c   1330]

MBUF state ACTIVE

DpMBufTypeMod: MBUF component UP (event=MBUF_DPEVT_UP)

DpMsgProcess: 3 server in MBUF

DpMsCheckServices()

DpMsCheckServices: send DpDelService(J2EE) to msgserver

Wed Sep  3 10:06:26 2014

DpModState: change server state from STARTING to ACTIVE

Wed Sep  3 10:08:11 2014

*** ERROR => DpHdlDeadWp: W6 (pid 6422714) died (severity=0, status=721035) [dpxxwp.c     1263]

Wed Sep  3 10:08:12 2014

DpTraceWpStatus: child (pid=6422714) killed with signal 11

Wed Sep  3 10:08:15 2014

********** SERVER SNAPSHOT 1 (Reason: Workprocess 6 died) - begin **********

Scheduler info

--------------

WP info

DpNumberOfDiaWps: dia_wps 9, standby_wps 0

  #dia     = 9

  #btc     = 3

  #standby = 0

  #max     = 22

General Scheduler info

  preemptionInfo.isActive           = true

  preemptionInfo.timeslice          = 500

  preemptionInfo.checkLoad          = true

Prio Class High

  maxRuntime[RQ_Q_PRIO_HIGH]     = 600 sec

  maxRuntimeHalf[RQ_Q_PRIO_HIGH] = 300 sec

Prio Class Normal

  maxRuntime[RQ_Q_PRIO_NORMAL]           = 3600 sec

  maxRuntimeHalf[RQ_Q_PRIO_NORMAL]       = 1800 sec

  maxTicketsForPrio[RQ_Q_PRIO_NORMAL]    = 8

  withPrioTickets[RQ_Q_PRIO_NORMAL]      = true

[Former Member · Accepted solution]

Try to update your kernel to latest patch level available and check again. Is errpt output in first message contains full information?

[former_member219483 · Accepted solution]

dear roman ,

i checked the kernel and i found the problem don't related to the sap kernel , the same resource group on app1 i moved it to another node app2  and the sap application run fine without any problem

[Sriram2009 · Accepted solution]

Hi Ahmed

You mean this is cluster fail over? still are you facing same issue ?

BR

SS

[former_member219483 · Accepted solution]

hi S Sriram ,

yes

[Sriram2009 · Accepted solution]

Hi

Oh, In that case both H/W not in same configurations?

I thing you can enable the ST12 trace to find the root case of Transaction FAGLB03

Regards

Ram

[former_member219483]

Dear All,

thanks for your support finally we solved this problem with open incident with sap support kindly find the replay from sap regarding to 7.40 kernel:

Note that all of the responsible function by the error are related with the database interface. This is a knonw issue with  db_fld_move function and there is a correction as from patch level 37 of kernel release 740.

we finally upgrade our kernel from 7.40 support pkg 12 to latest support pkg 60

thanks for Roman and s sriam for helping and support

thanks and BR

Ahmed Abd El-Aziz Ragab

[former_member219483]

Hi S Sriram ,

the 4 server have the same os Aix level version and sp.

app1:

root@ERPAPP1[/] > oslevel -s

7100-00-08-1241

app2 :

root@ERPAPP2[/] > oslevel -s

7100-00-08-1241

Db1:

root@ERPDB1[/] > oslevel -s

7100-00-08-1241

Db2:

root@ERPDB2[/] > oslevel -s

7100-00-08-1241

[Sriram2009]

Could you share the Dump?

[Sriram2009]

Could you share the Dump?

[former_member219483]

Dear all,

thanks for your respond .

i already checked kernel level of CI & the DI instance and i found this the same kernel level. our situation now we have 4 node for PRD environment , DB1 and BD2 working withe hacmp . the ora resource group now is online on db1 ie db1 contain CI and DB , the db2 contain now the instance  D03 .

for the two application server (app1 and app2 ) there is  a one resource group (app data contain /usr/sap/PRD/D04 , and there is a hacmp between app1 and app2 ,.

the problem happened when the resource group appdata was online on app1 and run a transaction on app1 the workprocess hanged and terminated , yesterday when i move the resource group appdata to the another node app2 the system working fine, from this action it's appear the problem not related to the sap kernel or the sap application , i think the problem related to the operating system on app1,

[Sriram2009]

Hi Ahmed

1.  For all the physical system are in AIX or different OS ?

2. As request the full Dump details?

Regards

[Former Member]

Please check dumps in ST22 at the same time as error occurred. Upload it if exists. Also try to generate stack call from the core file with dbx command.

[former_member188883]

Hi Ahmed,

Please share SAP application release and patch level.

Regards,

Deepak Kori

[former_member219483]

Dear Deepak,

thanks for your fast respond

the sap application release  7.40 patch level 02

Thanks&BR

Ahmed Ragab

[former_member188883]

Hi Ahmed,

As the issue appears in one of the dialog instance, please check whether you have same kernel release and patch across CI, and other dialog instances.

Dump from dialog instance may apprear due to mismatch of dispatcher files with other kernel files.

Regards,

Deepak Kori

[Sriram2009]

Hi Ahmed

Is this kernel level are different in CI & DI ?

BR

SS