Core file generated and system went down

Hi Guru's,

In work directory huge core file is generated application server went down. I deleted the core file and started the application it is working fine now.

(Abap Dump)ST22

SYSTEM_CORE_DUMPED

The SAP System work directoy(e.g./usr/sap/C11/D00/work) often contain a file called core

(System logs) SM21

Signal 33 receieved by operation System

Run-time error "SYSTEM_NO_ROLL" occured

Buffer SCSA Already exists(Lengh=4096)

Can you please let me know how to prevent this in future.

Regards,

Deve

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

Accepted Solutions (0)

Answers (3)

We can not prohibit the generation of core files of various sizes.Core files can be generated with various sizes.

We have to schedule a job/script in crontab for deletion of core files,when core file is created

Regards,

Sukarna

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

Hi Deve

you can schedule a job/script in crontab which will prohibit the generation of core files of more than 50MB or 02GB size (for eg.) at OS level.

Bhudev

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

Core files are generated becaus of program errors.

- Is this the first time that error occured?

- Which OS are you using?

- Which kernel patchlevel and database are you using?

- Is the problem reproducible?

Markus

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

Hi Markus,

Is this the first time that error occured:Ans:Second time we are facing the problem

- Which OS are you using:Ans:AIX

- Which kernel patchlevel and database are you using:640/247 and Oracle 9.2

- Is the problem reproducible:No

The core file size is more then 128gb .

Regards,

Deve

> The core file size is more then 128gb .

Usually, if a coredump is created, there also exists a corresponding dump (SYSTEM_CORE_DUMP) in ST22. If you have such a dump, check the "Active calls in SAP kernel" section and post it, there we may see, where the dump exactly occurs (in the SAP kernel or in the database client).

Most likely someone from SAP must analyze that further.

A coredump is the full content including memory of a process at the time it failed (http://en.wikipedia.org/wiki/Core_dump)

Markus

Hi Deve

May be you can utilize the idea as the following script what we have on our servers

kernel 700/166

OS AIX

oracle 10.2

r4lpar18:pr1adm 25> more remove_core.sh

#!/bin/ksh

This script is to be scheduled by cron under <SID>adm to run every x minutes.
and will find core files in the /usr/sap/<SID> directory and remove them

#----

. $HOME/.profile

If the user is <SID>adm then define PATH to be /usr/sap/<SID> directory
If the user is ora<SID> then define PATH to be $SAPDATA_HOME directory

if [ `/usr/bin/whoami | cut -c 4-6` = 'adm' ]

then

FINDPATH=/usr/sap/$SAPSYSTEMNAME/

elif [ `/usr/bin/whoami | cut -c 1-3` = 'ora' ]

then

FINDPATH=$SAPDATA_HOME # If the user is ora<SID> then look in the /oracle/<SID> directory

Search the path defined above

echo "`date` Path searching.... $FINDPATH"

for i in `find $FINDPATH -name "core" -type f`

echo "Execution date: " `date` "Core file: " $i

file $i

ls -l $i

if [ $? = 0 ]

then

rm $i

if [ $? = 0 ]

then

echo "File removed"

else

echo "Error deleting file!"

else

echo "File not found"

done

exit 0

Bhudev

Hi Markus,

Active calls in SAP kernel

=> 64 bit R/3 Kernel

=> 64 bit AIX Kernel

=> Heap limit = unlimited

=> Stack limit = unlimited

=> Core limit = 2147483648

=> File size limit = unlimited

=> Heap address = 0x0x1177ca100

=> Stack address = 0xfffffffffffa150

=> Stack low = 0xfffffffffff4540

=> Stack high = 0xffffffffffff950

=> Stack Trace:

bcopy() at 0x100005bd8

ImportComplexData1__FP8CONNE_RDPC12ImportOpHeadPCvPvUlUcPP5Stack() at 0x100ba7a30

ImportComplexData1__FP8CONNE_RDPC12ImportOpHeadPCvPvUlUcPP5Stack() at 0x100ba8fa0

ImportComplexData__FP8CONNE_RDPC12ImportOpHeadPCvPvUl() at 0x100ba0978

ab_connread__FUiP11IMPORT_INFOP23EXPIMP_DATA_OBJECTS_ADMP7AB_DATAPFPUsUlP23EXPIMP_DAT >

> A_OBJECTS_ADMPP7AB_DATAPPv_iPFPPUcPUlUi9EXPO_MODE_i() at 0x100b9fb44

expo_import__Fv() at 0x100bc37a8

ab_jimpo__Fv() at 0x100bc2fd4

ab_extri__Fv() at 0x10058c954

ab_xevent__FPCUs() at 0x1007a12dc

ab_dstep() at 0x100796b5c

dynpmcal() at 0x100c4c044

dynppbo0() at 0x100c497a4

dynprctl() at 0x100c517d0

dynpen00() at 0x100217680

Thdynpen00() at 0x1000a79a8

TskhLoop() at 0x1000ac56c

tskhstart() at 0x1000c2200

DpMain() at 0x1017c4f18

nlsui_main() at 0x101795b6c

=> CPU Registers:

msr = 0xa00000000000d0b2 iar = 0x0000000100005bd8

ctr = 0x0000000000000014 lr = 0x0000000100bb27a0

xer = 0x0000000020000012 cr = 0x0000000028282288

r00 = 0x0000000000000010 r01 = 0x0fffffffffffa150

r02 = 0x00000001110ef1b0 r03 = 0x0700000030be9e98

r04 = 0x0fffffffffffb3a4 r05 = 0x0000000000000041

r06 = 0x0700000030be9ff8 r07 = 0x0020002000200020

r08 = 0x0000000000000000 r09 = 0x0000000000000000

r10 = 0x0fffffffffffa0f0 r11 = 0x000000000000012c

r12 = 0x0000000100ba78c0 r13 = 0x0000000113964da0

r14 = 0x00000001109b7e80 r15 = 0x0000000110070030

r16 = 0x07000008f5aa4ce8 r17 = 0x07000008f5b078d8

r18 = 0x0000000000000000 r19 = 0x0700000030999c78

r20 = 0x0000000111112f60 r21 = 0x000000011111cdb8

r22 = 0x000000011016fc5c r23 = 0x000000011016fc58

r24 = 0x0000000000000a7c r25 = 0x0000000110804c38

r26 = 0x00000001018fba80 r27 = 0x0000000000000000

r28 = 0x0fffffffffffa860 r29 = 0x0fffffffffffac08

r30 = 0x0000000000000208 r31 = 0x0700000030be9e98

Regards,

Deve

Deve,

if you post such "fragments", please put next time code tags (the << >> sign in next to the underline) around the postings, this will make them much more readable and prevent the backend system from trying to "interpret" - thanx.

> bcopy() at 0x100005bd8

> ImportComplexData1__FP8CONNE_RDPC12ImportOpHeadPCvPvUlUcPP5Stack() at 0x100ba7a30

> ImportComplexData1__FP8CONNE_RDPC12ImportOpHeadPCvPvUlUcPP5Stack() at 0x100ba8fa0

> ImportComplexData__FP8CONNE_RDPC12ImportOpHeadPCvPvUl() at 0x100ba0978

> ab_connread__FUiP11IMPORT_INFOP23EXPIMP_DATA_OBJECTS_ADMP7AB_DATAPFPUsUlP23EXPIMP_DAT

So bcopy() was failing - this is an OS function (see man bcopy).

I would

- update the kernel to a recent version (patchlevel 276)

- check with note "Note 1119631 - Using SAP systems with AIX 5.3" if your OS is on a supported level (assuming you're runing 5.3, you didn't specify that)

Also as information: Oracle 9.2 is out of support.

Markus

Hi Markus,

Markus we are using Oracle 10.2.If we update the kernel to a recent version (patchlevel 276) this issue will be resolved. Please let me know.

Regards,

Deve

Please read my previous post.

I (and most probably nobody here) can tell if the error will go away by just installing a new kernel patch. The error happens in an operating system function so only SAP and/or IBM can tell.

Markus

Hi Markus,

I found below in workprocess trace in work directory

M stat = 4

M reqtype = 1

M act_reqtype = 1

M rq_info = 384

M tid = 323

M mode = 2

M len = 832

M rq_id = 32836

M rq_source = 1

M last_tid = 323

M last_mode = 2

M int_checked_resource(RFC) = 0

M ext_checked_resource(RFC) = 0

M int_checked_resource(HTTP) = 0

M ext_checked_resource(HTTP) = 0

M report = >ZTRGL_GL_COA_REPORT <

M action = 0

M tab_name = > <

M Modeinfo for User T323/M2

M tm state = 2

M uid = 27937

M term type = 0x4

M display = 0x8

M cpic_no = 1

M cpic_idx = 157

M usr = >PRIVERS <

M terminal = >wabelhdk0145867 <

M client = >810<

M conversation_ID = > <

M appc_tm_conv_idx = -1

M its_plugin = NO

M allowCreateMode = YES

M blockSoftCanel = NO

M imode = 1

M mode state = 0x42

M mode clean_state = 2

M task_type = ZTTADIA

M th_errno = 42

M rollout_reason = 1

M last_rollout_level = 7

M async_receives = 0

M cpic_receive = 0

M em handle = 194

M roll state = 4

M abap state = 4

M em state = 3

M eg state = 1

M spa state = 3

M enq state = 0

M softcancel = 1

M cancelInitiator = DISPATCHER

M next hook = T-1/U-1/M255

M master hook = T-1/U-1/M255

M slave hook = T-1/U-1/M255

M debug_tid = 255

M debug_mode = 0

M mode type = 0x1

M debug = 0

M tcode = >ZTFIGL10 <

M client conversation_ID = > <

M server conversation_ID = > <

M lock = 0

M max enq infos = 9

M act enq infos = 0

M em_hyper_hdl = NULL

M plugin_info = NULL

M act_plugin_hdl = -1

M act_plugin_no = 0

M max_plugin_no = 0

M Adresse Offset Data from APPC-CA-AREA

M -

M 0x70000002216be70 000000 060b0202 6d210000 009f0000 00000000 |....m!..........|

M 0x70000002216be80 000016 00ffffff ff000000 00000000 00001000 |................|

M 0x70000002216be90 000032 00000000 00000000 38373931 32343336 |........87912436|

M 0x70000002216bea0 000048 00000000 00000000 00000000 00000000 |................|

M 0x70000002216beb0 000064 00000000 00000000 00000000 009d007b |...............{|

M 0x70000002216bec0 000080 2a455252 2a003100 54685369 6748616e |ERR.1.ThSigHan|

M 0x70000002216bed0 000096 646c6572 3a207369 676e616c 00313100 |dler: signal.11.|

M 0x70000002216bee0 000112 5461736b 68616e64 6c657200 36343000 |Taskhandler.640.|

M 0x70000002216bef0 000128 31007468 78786865 61642e63 00393839 |1.thxxhead.c.989|

M 0x70000002216bf00 000144 33000057 65642041 70722020 31203136 |3..Wed Apr 1 16|

M 0x70000002216bf10 000160 3a31323a 34302032 30303900 00000037 |:12:40 2009....7|

M 0x70000002216bf20 000176 00534150 2d536572 76657220 70726f76 |.SAP-Server prov|

M 0x70000002216bf30 000192 70723035 5f565052 5f303020 6f6e2068 |pr05_VPR_00 on h|

M 0x70000002216bf40 000208 6f737420 70726f76 70723035 20287770 |ost provpr05 (wp|

M 0x70000002216bf50 000224 20313429 00000000 002a4552 522a0000 | 14).....ERR..|

M 0x70000002216bf60 000240 20002000 20002000 20002000 20002000 | . . . . . . . .|

M 0x70000002216bf70 000256 20002000 20002000 20002000 20002000 | . . . . . . . .|

M 0x70000002216bf80 000272 20002000 08000601 00007300 61007000 | . .......s.a.p.|

M 0x70000002216bf90 000288 63006f00 72006500 5f005600 50005200 |c.o.r.e._.V.P.R.|

M 0x70000002216bfa0 000304 5f003000 30000000 00000000 00000000 |_.0.0...........|

M 0x70000002216bfb0 000320 00000000 00000000 00000000 00000000 |................|

M 0x70000002216bfc0 000336 00000000 00000000 00000000 00000000 |................|

M 0x70000002216bfd0 000352 00000000 00000000 00000000 00000000 |................|

M 0x70000002216bfe0 000368 00000000 00000000 00000000 00000000 |................|

M 0x70000002216bff0 000384 00000000 00000000 00000000 00000000 |................|

M 0x70000002216c000 000400 00000000 00000000 00000000 00000000 |................|

M 0x70000002216c010 000416 00000000 00000000 00000000 00000000 |................|

M 0x70000002216c020 000432 00000000 00000000 00000000 00000000 |................|

M 0x70000002216c030 000448 00000000 00000000 00000000 00000000 |................|

M 0x70000002216c040 000464 00000000 00000000 00000000 00000000 |................|

M 0x70000002216c050 000480 00000000 00000000 00000000 00000000 |................|

M 0x70000002216c060 000496 00000000 00000000 00000000 00000000 |................|

M -

M PfStatDisconnect: disconnect statistics

M Entering ThSetStatError

M ThIErrHandle: don't try rollback again

M Entering ThReadDetachMode

M *** ERROR => ThIErrHandle: bad value for th_act_em_hdl (194), detach T323/M2 [thxxhead.c 10141]

M Modeinfo for User T323/M2

M tm state = 2

M uid = 27937

M term type = 0x4

M display = 0x8

M cpic_no = 1

M cpic_idx = 157

M usr = >PRIVERS <

M terminal = >wabelhdk0145867 <

M client = >810<

M conversation_ID = > <

M appc_tm_conv_idx = -1

M its_plugin = NO

M allowCreateMode = YES

M blockSoftCanel = NO

M imode = 1

M mode state = 0x42

M mode clean_state = 2

M task_type = ZTTADIA

M th_errno = 42

M rollout_reason = 1

M last_rollout_level = 7

M async_receives = 0

M cpic_receive = 0

M em handle = 194

M roll state = 4

M abap state = 4

M em state = 2