cancel
Showing results for 
Search instead for 
Did you mean: 

BW System crashed due to DB6_ADM_RUNSTATS, error -911, -1032, --1224

former_member286941
Participant
0 Kudos

Hi Friends,

we are using AIX 5.2 with DB2 8.1 on BW 3.5/640 system.

The below warning is also coming very frequently which is causing system crash sometimes:

1.Function DB6_ADM_RUNSTATS failed for List of Tables . Reason: ADBC

2. Error and Database error -911 at EXE

3. BTC 029 061 SEMABL BYL Database error -1032

4. DIA 003 061 SEJABE BYM SQL error -1224 occurred;

work process in reconnect status

18:58:59 DP Q04 Connection to user

14871 (ITGARO ), terminal 125 (nbw20810 ) lost

19:04:06 BTC 029 061 SEMABL BYL Database error -1032

requires intervention from database administrator

19:04:06 BTC 029 061 SEMABL BY4 Database error -1032 at PRE

access to table SWT_LOGCAT

19:04:06 BTC 029 061 SEMABL BY0 > SQL1032N No start database

manager command was issued.

19:04:06 DIA 012 061 USEDBA BY2 Database error -1032 at CON

19:04:06 DIA 017 061 DEKASO BY2 Database error -1032 at CON

19:04:06 DIA 014 R19 Initialization DB-

Connect Failed, Return Code 000256

19:04:06 DIA 019 061 CHPAPR BYL Database error -1032

requires intervention from database administrator

19:04:46 DIA 003 061 SEJABE BYM SQL error -1224 occurred;

work process in reconnect status

19:04:46 DIA 012 061 USEDBA BYM SQL error -1224 occurred;

work process in reconnect status

19:04:46 DIA 005 061 DKKILA BYM SQL error -1224 occurred;

work process in reconnect status

19:04:46 DIA 015 061 NLBAGR BYM SQL error -1224 occurred;

work process in reconnect status

19:04:46 BTC 030 000 SESYSTADMJOB BY2 Database error -1224 at OPC

19:04:46 BTC 030 000 SESYSTADMJOB BY0 > SQL1224N The database

manager is not able to accept new

19:04:46 BTC 030 000 SESYSTADMJOB BY0 > requests, has terminated

all requests in progress, or has

19:04:46 BTC 030 000 SESYSTADMJOB BY0 > terminated your particular

request due to a problem with you

19:04:46 BTC 030 000 SESYSTADMJOB BY0 > request. SQLSTATE=55032

19:04:46 DIA 017 061 DEKASO BYM SQL error -1224 occurred;

work process in reconnect status

19:04:46 DIA 017 061 DEKASO R38 Error at DB commit, return

code 001024

any idea what can be done to resolve this issue.

Thanks and regards

Regards - Munish

Accepted Solutions (0)

Answers (11)

Answers (11)

former_member286941
Participant
0 Kudos

not-helpful.

former_member286941
Participant
0 Kudos

My db2 versioni is DB2 v9.1.0.1

I was mistaken.

Former Member
0 Kudos

Munish,

The error "sqlerReturnFmpToPool, probe:910 MESSAGE : marking fmp as unstable" even though marked as a "severe" message is a non-fatal error and was corrected in subsequent releases of DB2. This can be fixed (temporarily) by setting KEEPFENCED to YES and FENCED_POOL to a small value number (5 or 10). However, you can always ignore this particular error as it does not cause any problems.

Now as far as your database crash is concerned, its most likely related to a particular APAR that you might have hit since you are running on an older release of DB2 version 9.1. There are several APARs related to instance problems (or crash) caused by runstats (or runstats which make use of profile). We had a similar problem in our environment where the instance used to crash as soon as runstats was issued on certain tables and this ended up being a new APAR (IZ41060) which was fixed in a subsequent fixpack. The local fix in this situation was to issue:

db2 runstats on table sap<sid>.<tablename> and indexes all allow write access

This command will update the stats and clear the stats profile and was required to be executed on every table in our environment. This is an online activity and you should be able to run this while the system is up.

Hope this helps..

Regards,

Sameer

former_member286941
Participant
0 Kudos

Sort heap thres for shared sorts (4KB) (SHEAPTHRES_SHR) = AUTOMATIC

STMM is not found in parameters.

Permissions are ok now. i am able to use db6dutil now, corrected the password problem.

i have no idea regarding db2diag -A, our db2 guy might be known to this.

Pls let me know if anything to resolve this issue.

best regards,

Munish

Former Member
0 Kudos

Please have a look at oss note 175036 to create an dbsl deadlock trace. The command db2diag -A you can schedule on you operating system e.g. for every day at the same hour to minimize the size of the diag.log. The user is db2<SID> which runs the command.

If you use stmm you should have set the following parameters.

SELF_TUNING_MEM = ON [STMM=YES]

DATABASE_MEMORY >= 100000 [STMM=YES]

LOCKLIST = AUTOMATIC [STMM=YES]

MAXLOCKS = AUTOMATIC [STMM=YES]

PCKCACHESZ = AUTOMATIC [STMM=YES]

SHEAPTHRES_SHR = AUTOMATIC [STMM=YES]

SORTHEAP = AUTOMATIC [STMM=YES]

Regards

Olaf

Edited by: Olaf Balzer on Jan 14, 2010 2:11 PM

former_member286941
Participant
0 Kudos

tail -310 db2diag.log

/db2/BWP/log_dir/NODE0000/.

2010-01-14-10.32.03.583933+060 I15733311A444 LEVEL: Event

PID : 798956 TID : 1 PROC : db2stmm (BWP) 0

INSTANCE: db2bwp NODE : 000 DB : BWP

APPHDL : 0-52 APPID: *LOCAL.db2bwp.100111182342

AUTHID : SAPBWP

FUNCTION: DB2 UDB, config/install, sqlfLogUpdateCfgParam, probe:20

CHANGE : STMM CFG DB BWP: "Sheapthres_shr" From: "73056" <automatic> To: "81793" <automatic>

2010-01-14-10.32.03.593671+060 I15733756A436 LEVEL: Event

PID : 798956 TID : 1 PROC : db2stmm (BWP) 0

INSTANCE: db2bwp NODE : 000 DB : BWP

APPHDL : 0-52 APPID: *LOCAL.db2bwp.100111182342

AUTHID : SAPBWP

FUNCTION: DB2 UDB, config/install, sqlfLogUpdateCfgParam, probe:20

CHANGE : STMM CFG DB BWP: "Sortheap" From: "3653" <automatic> To: "4089" <automatic>

2010-01-14-10.42.55.641172+060 I15734193A271 LEVEL: Warning

PID : 3633166 TID : 1

FUNCTION: DB2 Common, Security, Users and Groups, secLogMessage, probe:20

DATA #1 : String, 65 bytes

Password validation for user bwpadm failed with rc = -2146500507

2010-01-14-10.42.55.753060+060 I15734465A271 LEVEL: Warning

PID : 3633168 TID : 1

FUNCTION: DB2 Common, Security, Users and Groups, secLogMessage, probe:20

DATA #1 : String, 65 bytes

Password validation for user bwpadm failed with rc = -2146500507

2010-01-14-10.42.56.076470+060 I15734737A271 LEVEL: Warning

PID : 3633170 TID : 1

FUNCTION: DB2 Common, Security, Users and Groups, secLogMessage, probe:20

DATA #1 : String, 65 bytes

Password validation for user bwpadm failed with rc = -2146500507

2010-01-14-10.42.57.940361+060 I15735009A271 LEVEL: Warning

PID : 3633172 TID : 1

FUNCTION: DB2 Common, Security, Users and Groups, secLogMessage, probe:20

DATA #1 : String, 65 bytes

Password validation for user bwpadm failed with rc = -2146500507

2010-01-14-10.42.57.984990+060 I15735281A271 LEVEL: Warning

PID : 3633172 TID : 1

FUNCTION: DB2 Common, Security, Users and Groups, secLogMessage, probe:20

DATA #1 : String, 65 bytes

Password validation for user bwpadm failed with rc = -2146500507

2010-01-14-10.42.58.227268+060 I15735553A271 LEVEL: Warning

PID : 3633174 TID : 1

FUNCTION: DB2 Common, Security, Users and Groups, secLogMessage, probe:20

DATA #1 : String, 65 bytes

Password validation for user bwpadm failed with rc = -2146500507

2010-01-14-10.51.33.683730+060 I15735825A462 LEVEL: Info

PID : 798956 TID : 1 PROC : db2stmm (BWP) 0

INSTANCE: db2bwp NODE : 000 DB : BWP

APPHDL : 0-52 APPID: *LOCAL.db2bwp.100111182342

AUTHID : SAPBWP

FUNCTION: DB2 UDB, buffer pool services, sqlbAlterBufferPoolAct, probe:90

MESSAGE : Altering bufferpool "BP_STD_16K" From: "1348672" <automatic> To:

"1356492" <automatic>

2010-01-14-10.53.03.721301+060 I15736288A567 LEVEL: Warning

PID : 798956 TID : 1 PROC : db2stmm (BWP) 0

INSTANCE: db2bwp NODE : 000 DB : BWP

APPHDL : 0-52 APPID: *LOCAL.db2bwp.100111182342

AUTHID : SAPBWP

FUNCTION: DB2 UDB, Self tuning memory manager, stmmUpdateDBConfig, probe:285

MESSAGE : ZRC=0x82AE00A6=-2102525786=STMM_DYNAMIC_UPDATE_FAILED

"STMM dynamic update failed"

DATA #1 : String, 76 bytes

Dynamic update failed for parameter Locklist. Attempted update value is 5472

2010-01-14-10.53.03.722593+060 I15736856A444 LEVEL: Event

PID : 798956 TID : 1 PROC : db2stmm (BWP) 0

INSTANCE: db2bwp NODE : 000 DB : BWP

APPHDL : 0-52 APPID: *LOCAL.db2bwp.100111182342

AUTHID : SAPBWP

FUNCTION: DB2 UDB, config/install, sqlfLogUpdateCfgParam, probe:20

CHANGE : STMM CFG DB BWP: "Sheapthres_shr" From: "81793" <automatic> To: "78415" <automatic>

2010-01-14-10.53.04.168179+060 I15737301A458 LEVEL: Info

PID : 798956 TID : 1 PROC : db2stmm (BWP) 0

INSTANCE: db2bwp NODE : 000 DB : BWP

APPHDL : 0-52 APPID: *LOCAL.db2bwp.100111182342

AUTHID : SAPBWP

FUNCTION: DB2 UDB, buffer pool services, sqlbAlterBufferPoolAct, probe:90

MESSAGE : Altering bufferpool "IBMDEFAULTBP" From: "6524" <automatic> To:

"5244" <automatic>

2010-01-14-10.53.04.182145+060 I15737760A436 LEVEL: Event

PID : 798956 TID : 1 PROC : db2stmm (BWP) 0

INSTANCE: db2bwp NODE : 000 DB : BWP

APPHDL : 0-52 APPID: *LOCAL.db2bwp.100111182342

AUTHID : SAPBWP

FUNCTION: DB2 UDB, config/install, sqlfLogUpdateCfgParam, probe:20

CHANGE : STMM CFG DB BWP: "Sortheap" From: "4089" <automatic> To: "3920" <automatic>

2010-01-14-10.57.13.883762+060 I15738197A271 LEVEL: Warning

PID : 3522744 TID : 1

FUNCTION: DB2 Common, Security, Users and Groups, secLogMessage, probe:20

DATA #1 : String, 65 bytes

Password validation for user bwpadm failed with rc = -2146500507

2010-01-14-10.57.14.048828+060 I15738469A271 LEVEL: Warning

PID : 3522746 TID : 1

FUNCTION: DB2 Common, Security, Users and Groups, secLogMessage, probe:20

DATA #1 : String, 65 bytes

Password validation for user bwpadm failed with rc = -2146500507

2010-01-14-10.57.14.216675+060 I15738741A271 LEVEL: Warning

PID : 3522746 TID : 1

FUNCTION: DB2 Common, Security, Users and Groups, secLogMessage, probe:20

DATA #1 : String, 65 bytes

Password validation for user bwpadm failed with rc = -2146500507

2010-01-14-10.57.14.285737+060 I15739013A271 LEVEL: Warning

PID : 3522746 TID : 1

FUNCTION: DB2 Common, Security, Users and Groups, secLogMessage, probe:20

DATA #1 : String, 65 bytes

Password validation for user bwpadm failed with rc = -2146500507

2010-01-14-10.57.14.917712+060 I15739285A271 LEVEL: Warning

PID : 3522746 TID : 1

FUNCTION: DB2 Common, Security, Users and Groups, secLogMessage, probe:20

DATA #1 : String, 65 bytes

Password validation for user bwpadm failed with rc = -2146500507

2010-01-14-10.57.14.980479+060 I15739557A271 LEVEL: Warning

PID : 3522746 TID : 1

FUNCTION: DB2 Common, Security, Users and Groups, secLogMessage, probe:20

DATA #1 : String, 65 bytes

Password validation for user bwpadm failed with rc = -2146500507

2010-01-14-10.57.15.254151+060 I15739829A271 LEVEL: Warning

PID : 3522746 TID : 1

FUNCTION: DB2 Common, Security, Users and Groups, secLogMessage, probe:20

DATA #1 : String, 65 bytes

Password validation for user bwpadm failed with rc = -2146500507

2010-01-14-10.57.15.308542+060 I15740101A271 LEVEL: Warning

PID : 3522746 TID : 1

FUNCTION: DB2 Common, Security, Users and Groups, secLogMessage, probe:20

DATA #1 : String, 65 bytes

Password validation for user bwpadm failed with rc = -2146500507

2010-01-14-10.57.15.586406+060 I15740373A271 LEVEL: Warning

PID : 3522746 TID : 1

FUNCTION: DB2 Common, Security, Users and Groups, secLogMessage, probe:20

DATA #1 : String, 65 bytes

Password validation for user bwpadm failed with rc = -2146500507

2010-01-14-10.57.15.650049+060 I15740645A271 LEVEL: Warning

PID : 3522746 TID : 1

FUNCTION: DB2 Common, Security, Users and Groups, secLogMessage, probe:20

DATA #1 : String, 65 bytes

Password validation for user bwpadm failed with rc = -2146500507

2010-01-14-10.57.15.934485+060 I15740917A271 LEVEL: Warning

PID : 3522746 TID : 1

FUNCTION: DB2 Common, Security, Users and Groups, secLogMessage, probe:20

DATA #1 : String, 65 bytes

Password validation for user bwpadm failed with rc = -2146500507

2010-01-14-10.57.16.023114+060 I15741189A271 LEVEL: Warning

PID : 3522746 TID : 1

FUNCTION: DB2 Common, Security, Users and Groups, secLogMessage, probe:20

DATA #1 : String, 65 bytes

Password validation for user bwpadm failed with rc = -2146500507

2010-01-14-10.57.16.289905+060 I15741461A271 LEVEL: Warning

PID : 3522746 TID : 1

FUNCTION: DB2 Common, Security, Users and Groups, secLogMessage, probe:20

DATA #1 : String, 65 bytes

Password validation for user bwpadm failed with rc = -2146500507

2010-01-14-10.57.16.345930+060 I15741733A271 LEVEL: Warning

PID : 3522746 TID : 1

FUNCTION: DB2 Common, Security, Users and Groups, secLogMessage, probe:20

DATA #1 : String, 65 bytes

Password validation for user bwpadm failed with rc = -2146500507

2010-01-14-10.57.16.627395+060 I15742005A271 LEVEL: Warning

PID : 3522746 TID : 1

FUNCTION: DB2 Common, Security, Users and Groups, secLogMessage, probe:20

DATA #1 : String, 65 bytes

Password validation for user bwpadm failed with rc = -2146500507

2010-01-14-10.57.16.701147+060 I15742277A271 LEVEL: Warning

PID : 3522746 TID : 1

FUNCTION: DB2 Common, Security, Users and Groups, secLogMessage, probe:20

DATA #1 : String, 65 bytes

Password validation for user bwpadm failed with rc = -2146500507

2010-01-14-10.57.31.076427+060 I15742549A379 LEVEL: Severe

PID : 786644 TID : 1 PROC : db2sysc 0

INSTANCE: db2bwp NODE : 000

FUNCTION: DB2 UDB, routine_infrastructure, sqlerReturnFmpToPool, probe:910

MESSAGE : marking fmp as unstable:

DATA #1 : Hexdump, 4 bytes

0x078000000041EF40 : 0036 3064 .60d

2010-01-14-10.57.31.076634+060 I15742929A2815 LEVEL: Severe

PID : 786644 TID : 1 PROC : db2sysc 0

INSTANCE: db2bwp NODE : 000

FUNCTION: DB2 UDB, routine_infrastructure, sqlerReturnFmpToPool, probe:911

MESSAGE : marking fmp as unstable:

DATA #1 : Hexdump, 488 bytes

0x078000000041EF40 : 0036 3064 0000 0000 018A 0000 0000 0000 .60d............

0x078000000041EF50 : 018B 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041EF60 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041EF70 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041EF80 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041EF90 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041EFA0 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041EFB0 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041EFC0 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041EFD0 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041EFE0 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041EFF0 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041F000 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041F010 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041F020 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041F030 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041F040 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041F050 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041F060 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041F070 : 0000 0000 0000 0000 0000 0001 111F 7720 ..............w

0x078000000041F080 : 0780 0000 1007 0094 0780 0000 1008 0094 ................

0x078000000041F090 : 0001 0000 0000 0000 0000 0000 0000 029A ................

0x078000000041F0A0 : 0000 00A3 0000 0000 0000 0000 0000 0000 ................

0x078000000041F0B0 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041F0C0 : 0780 0000 0023 AB20 0780 0000 0041 EF40 .....#. .....A.@

0x078000000041F0D0 : 0780 0000 002C FF00 0000 0001 0000 0001 .....,..........

0x078000000041F0E0 : 0000 0000 0000 0000 0000 0001 0000 0000 ................

0x078000000041F0F0 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041F100 : 0780 0000 001A FB60 0000 0000 0000 0000 .......`........

0x078000000041F110 : 0000 0150 0000 0000 0000 0000 0000 0001 ...P............

0x078000000041F120 : 0100 0000 0000 0000 ........

2010-01-14-10.57.31.076872+060 I15745745A426 LEVEL: Warning

PID : 548998 TID : 1 PROC : db2agent (BWP) 0

INSTANCE: db2bwp NODE : 000 DB : BWP

APPHDL : 0-28 APPID: *LOCAL.db2bwp.100111182348

AUTHID : SAPBWP

MESSAGE : Removing FMP from pool

DATA #1 : Hexdump, 16 bytes

0x0FFFFFFFFFFF0748 : 0780 0000 0041 EF58 0036 3064 0000 0000 .....A.X.60d....

2010-01-14-10.57.34.238172+060 I15746172A444 LEVEL: Event

PID : 798956 TID : 1 PROC : db2stmm (BWP) 0

INSTANCE: db2bwp NODE : 000 DB : BWP

APPHDL : 0-52 APPID: *LOCAL.db2bwp.100111182342

AUTHID : SAPBWP

FUNCTION: DB2 UDB, config/install, sqlfLogUpdateCfgParam, probe:20

CHANGE : STMM CFG DB BWP: "Sheapthres_shr" From: "78415" <automatic> To: "73992" <automatic>

2010-01-14-10.57.34.241586+060 I15746617A436 LEVEL: Event

PID : 798956 TID : 1 PROC : db2stmm (BWP) 0

INSTANCE: db2bwp NODE : 000 DB : BWP

APPHDL : 0-52 APPID: *LOCAL.db2bwp.100111182342

AUTHID : SAPBWP

FUNCTION: DB2 UDB, config/install, sqlfLogUpdateCfgParam, probe:20

CHANGE : STMM CFG DB BWP: "Sortheap" From: "3920" <automatic> To: "3699" <automatic>

2010-01-14-10.57.40.592365+060 I15747054A271 LEVEL: Warning

PID : 3522628 TID : 1

FUNCTION: DB2 Common, Security, Users and Groups, secLogMessage, probe:20

DATA #1 : String, 65 bytes

Password validation for user bwpadm failed with rc = -2146500507

];bwpadm@nbcsaprtv016:/db2/BWP/db2dump

Former Member
0 Kudos

Do you have a look at the permission warnings for user bwpadm? Also you use STMM do you have the parameter SHEAPTHRES_SHR set to automatic? Are you planning a database upgrade because the fixpack 1 is very old?

Are you getting the same four errors often in the same order? Have you scheduled the command db2diag -A to switch and archive the db2diag.log from time to time because an analyze in a big file is a little bit hard.

Could you also please check whether you get abap short dumps for the relevant time slot? What is your value for parameter locklist? You should also create a dbsl deadlock trace and at the same time, during the deadlock situation, a db6util output.

Regards

Olaf

Edited by: Olaf Balzer on Jan 14, 2010 1:33 PM

Edited by: Olaf Balzer on Jan 14, 2010 1:44 PM

Edited by: Olaf Balzer on Jan 14, 2010 1:59 PM

former_member286941
Participant
0 Kudos

tail -279 8 dev _w0| wc -

M ThISend: complete stat record

M ThSendToLocGw: send sap wp rq to local gateway

M -OUT- tid 113 uid 14792 mode 0

M -OUT- appc_ca_blk 95 len 80 rq_id 60294

M -OUT- wait_f_answ 0 cmd 0

M

M NiIWrite: write 28, 1 packs, MESG_IO, hdl 5, data complete

M ThISend: th_wp_waits_for_rq = TH_WP_WAITS_FOR_RQ

M ThISend: soft cancel for T113/M0 (name=SEANBR ) finished

M DpSemRq: key: 23, units: 1, timeout: -1

M DpSemRel: key: 23, units: 1

M DpRqPutIntoQueue: put request into queue (reqtype 0, prio LOW, rq_id 60293)

M -OUT- sender_id WORK_PROCESS tid 113 wp_ca_blk -1 wp_id 0

M -OUT- action SEND_TO_APPCTM uid 14792 appc_ca_blk -1 type NOWP

M -OUT- new_stat WP_WAIT mode 0 len 0 rq_id 60293

M nihsl-getHostAddr: found hostname 'localhost' in cache

M nihsi-getHostAddr: hostname 'localhost' = addr 127.0.0.1

M nihsl-getServNo: got servicename 'sapdp22' from operating system

M nihsi-getServNo: servicename 'sapdp22' = port 0C.96/3222

M NiIInitSocket: set default settings for socket 12

M NiCreateHandle: state hdl 4 / socket 12 NI_INITIAL

M NiIDgSend: init datagram send hdl 4 / socket 12

M NiIDgSend: connect dgram to: host 127.0.0.1, port 0C.96/3222, fam 2 (low adr..high adr)

M NiIDgsend: sending on connected datagram-handle

M REL WP ca_blk 1

M SosGetAnchor: get anchor for 13

M PfRollOut: roll out tf/l_et(s,us)/l_rt/l_h/luw = 3600/(0,0)/0/0/0x2/3

M PfRollOut: short roll out completed

M ThIRollOut: roll out T113/U14792/M0 (level=7, short_roll_out=0)

M ThCallHooks: call hook >SAP-Trace RFC save< for event BEFORE_ROLL_OUT

M TrThHookFunc: called for Rollout

M ThCallHooks: call hook >dyRollInOutHook< for event BEFORE_ROLL_OUT

M ThCallHooks: call hook >ITSP: RollOut< for event BEFORE_ROLL_OUT

M SosSearchAnchor: search anchor for 16

M ThCallHooks: call hook >RtmRollOut< for event BEFORE_ROLL_OUT

M ThCallHooks: call hook >rsts_before_rollout< for event BEFORE_ROLL_OUT

M SosSearchAnchor: search anchor for 10

M SosSearchAnchor: search anchor for 10

M ThIRollOut: call rrol_out (1)

M ThIRollOut: act roll state = DP_ROLLED_OUT

M ThIRollOut: roll level <> 0, don't call ab_rollout

M ThIRollOut: full roll out of U14792 M0 I0 (Level 7) ok

M ThISend: save spa pointer 0x113fe5890 and install dummy spa

M ThISend: save snc contexts

M ThISncSaveAllContexts: save snc contexts

M ThCheckEmMagic: check em magic at 0x700000060101450 for T113/M0/I0

M ThCheckEmMagic: em magic at 0x700000060101450 for T113/M0/I0 o.k.

M ThEmContextDetach2: detach T113/M0 from em memory (em_hdl=0, force=0)

M ThEmContextDetach2: reset local em info

M ThCallHooks: call hook >SAP-Trace buffer write< for event BEFORE_TASK_SWITCH

M TrThHookFunc: called for task switch

M ThCallHooks: call hook >ImcResetPtr< for event BEFORE_TASK_SWITCH

M ThCallHooks: call hook >HttpResetPtr< for event BEFORE_TASK_SWITCH

M ThCallHooks: call hook >DbgBeforeTaskSwitch< for event BEFORE_TASK_SWITCH

M save task type/act rqtype 0/1 of T113/M0

M TskhLoop: switch back to normal tracel level 1

M extStatHook: This process does not report performance statistics

B

B Thu Jan 14 04:45:09 2010

B table logging switched off for all clients

S

S Thu Jan 14 04:46:09 2010

S found spool memory service RSPO-ACTIONS at 0x7000004c65ec040

M

M Thu Jan 14 04:48:59 2010

M SecAudit(RsauShmInit): WP attached to existing shared memory.

M SecAudit(RsauShmInit): addr of SCSA........... = 0x700000030000000

M SecAudit(RsauShmInit): addr of RSAUSHM........ = 0x700000030000768

M SecAudit(RsauShmInit): addr of RSAUSLOTINFO... = 0x7000000300007a0

M SecAudit(RsauShmInit): addr of RSAUSLOTS...... = 0x7000000300007ac

N

N Thu Jan 14 04:54:59 2010

N login/password_change_for_SSO : 1 -> 1

B

B Thu Jan 14 04:55:02 2010

B create_con (con_name=R/3*)

B New connection 1 created

C DB2 client driver version '09.01.0001'

C Connected to DB2 server type 'DB2/AIX64'

C Connected to DB2 version '09.01.0001'

C Connect to 'BWP' as 'SAPBWP' schema 'SAPBWP' o.k.; con_hdl=1

C DbSlControl: returning SAPDBHOST='nbcsapcph016'

B Connection 1 opened (DBSL handle 1)

B Wp Hdl ConName ConId ConState TX PRM RCT TIM MAX OPT Date Time DBHost

B 000 000 R/3 000000000 ACTIVE NO YES NO 000 255 255 20100114 044504 nbcsapcph016

B 000 001 R/3* 000002295 ACTIVE NO NO NO 004 255 255 20100114 045502 nbcsapcph016

M

M Thu Jan 14 04:58:02 2010

M ThSwitchSession:: th_enforce_rollout = 0

B

B Thu Jan 14 05:00:29 2010

B dbmyclu : info : my major identification is 171858960, minor one 22.

B dbmyclu : info : Time Reference is 1.12.2001 00:00:00h GMT.

B dbmyclu : info : my uuid is 4B4E953693430130E10080000A3E5C10.

B dbmyclu : info : current optimistic cluster level: 2

B dbmyclu : info : pessimistic reads set to 2.

C

C Thu Jan 14 06:39:58 2010

C disconnected from 'BWP', con_hdl=1

B Disconnected from connection con_da={R/3*,2334,1}

C

C Thu Jan 14 06:43:34 2010

C DB2 client driver version '09.01.0001'

C Connected to DB2 server type 'DB2/AIX64'

C Connected to DB2 version '09.01.0001'

C Connect to 'BWP' as 'SAPBWP' schema 'SAPBWP' o.k.; con_hdl=1

B Connection 1 opened (DBSL handle 1)

B Wp Hdl ConName ConId ConState TX PRM RCT TIM MAX OPT Date Time DBHost

B 000 000 R/3 000000000 ACTIVE NO YES NO 000 255 255 20100114 044504 nbcsapcph016

B 000 001 R/3* 000002335 ACTIVE NO NO NO 004 255 255 20100114 064333 nbcsapcph016

B

B Thu Jan 14 06:48:29 2010

B create_con (con_name=R/3*TIMESTAMP)

B New connection 2 created

M

M Thu Jan 14 06:52:03 2010

M ***LOG R2G=> ThPlgTimeout, Timeout () [thxxplg.c 5156]

C

C Thu Jan 14 07:00:47 2010

C disconnected from 'BWP', con_hdl=1

B Disconnected from connection con_da={R/3*,2360,1}

M

M Thu Jan 14 07:32:09 2010

M *** WARNING => big time diff 1881743466 [2613422885 200199055]for stamp STAT_READDIR, correct ???

C

C Thu Jan 14 07:35:27 2010

C DB2 client driver version '09.01.0001'

C Connected to DB2 server type 'DB2/AIX64'

C Connected to DB2 version '09.01.0001'

C Connect to 'BWP' as 'SAPBWP' schema 'SAPBWP' o.k.; con_hdl=1

B Connection 1 opened (DBSL handle 1)

B Wp Hdl ConName ConId ConState TX PRM RCT TIM MAX OPT Date Time DBHost

B 000 000 R/3 000000000 ACTIVE NO YES NO 000 255 255 20100114 044504 nbcsapcph016

B 000 001 R/3* 000002361 ACTIVE NO NO NO 004 255 255 20100114 073527 nbcsapcph016

B 000 002 R/3*TIMESTAMP 000002356 DISCONNECTED NO NO NO 003 255 255 20100114 065617 nbcsapcph016

M

M Thu Jan 14 07:57:07 2010

M *** WARNING => PfSetGuiTime: gui time too large, discard (gui=513984/resptime=188417)

M

M Thu Jan 14 08:16:04 2010

M ***LOG R2G=> ThPlgTimeout, Timeout () [thxxplg.c 5156]

M

M Thu Jan 14 09:00:15 2010

M ThIUsrDel: th_rollback_usrdelentry = 1

M

M Thu Jan 14 09:11:25 2010

M *** WARNING => PfSetGuiTime: gui time too large, discard (gui=839735/resptime=133766)

M

M Thu Jan 14 10:08:05 2010

M ***LOG R2G=> ThPlgTimeout, Timeout () [thxxplg.c 5156]

C

C Thu Jan 14 10:21:17 2010

C disconnected from 'BWP', con_hdl=1

B Disconnected from connection con_da={R/3*,2443,1}

C

C Thu Jan 14 10:21:48 2010

C DB2 client driver version '09.01.0001'

C Connected to DB2 server type 'DB2/AIX64'

C Connected to DB2 version '09.01.0001'

C Connect to 'BWP' as 'SAPBWP' schema 'SAPBWP' o.k.; con_hdl=1

B Connection 1 opened (DBSL handle 1)

B Wp Hdl ConName ConId ConState TX PRM RCT TIM MAX OPT Date Time DBHost

B 000 000 R/3 000000000 ACTIVE NO YES NO 000 255 255 20100114 044504 nbcsapcph016

B 000 001 R/3* 000002444 ACTIVE NO NO NO 004 255 255 20100114 102148 nbcsapcph016

B 000 002 R/3*TIMESTAMP 000002399 DISCONNECTED NO NO NO 004 255 255 20100114 085006 nbcsapcph016

M

M Thu Jan 14 10:37:15 2010

M *** WARNING => PfSetGuiTime: gui time too large, discard (gui=3731359/resptime=194176)

M

M Thu Jan 14 10:46:19 2010

M *** WARNING => PfSetGuiTime: gui time too large, discard (gui=518719/resptime=68566)

C

C Thu Jan 14 10:48:32 2010

C disconnected from 'BWP', con_hdl=1

B Disconnected from connection con_da={R/3*,2451,1}

C DB2 client driver version '09.01.0001'

C Connected to DB2 server type 'DB2/AIX64'

C Connected to DB2 version '09.01.0001'

C Connect to 'BWP' as 'SAPBWP' schema 'SAPBWP' o.k.; con_hdl=1

B Connection 1 opened (DBSL handle 1)

B Wp Hdl ConName ConId ConState TX PRM RCT TIM MAX OPT Date Time DBHost

B 000 000 R/3 000000000 ACTIVE YES YES NO 000 255 255 20100114 044504 nbcsapcph016

B 000 001 R/3* 000002452 ACTIVE NO NO NO 004 255 255 20100114 104832 nbcsapcph016

B 000 002 R/3*TIMESTAMP 000002399 DISCONNECTED NO NO NO 004 255 255 20100114 085006 nbcsapcph016

C disconnected from 'BWP', con_hdl=1

B Disconnected from connection con_da={R/3*,2452,1}

C

C Thu Jan 14 10:48:33 2010

C DB2 client driver version '09.01.0001'

C Connected to DB2 server type 'DB2/AIX64'

C Connected to DB2 version '09.01.0001'

C Connect to 'BWP' as 'SAPBWP' schema 'SAPBWP' o.k.; con_hdl=1

B Connection 1 opened (DBSL handle 1)

B Wp Hdl ConName ConId ConState TX PRM RCT TIM MAX OPT Date Time DBHost

B 000 000 R/3 000000000 ACTIVE YES YES NO 000 255 255 20100114 044504 nbcsapcph016

B 000 001 R/3* 000002453 ACTIVE NO NO NO 004 255 255 20100114 104833 nbcsapcph016

B 000 002 R/3*TIMESTAMP 000002399 DISCONNECTED NO NO NO 004 255 255 20100114 085006 nbcsapcph016

C disconnected from 'BWP', con_hdl=1

B Disconnected from connection con_da={R/3*,2453,1}

C

C Thu Jan 14 10:48:34 2010

C DB2 client driver version '09.01.0001'

C Connected to DB2 server type 'DB2/AIX64'

C Connected to DB2 version '09.01.0001'

C Connect to 'BWP' as 'SAPBWP' schema 'SAPBWP' o.k.; con_hdl=1

B Connection 1 opened (DBSL handle 1)

B Wp Hdl ConName ConId ConState TX PRM RCT TIM MAX OPT Date Time DBHost

B 000 000 R/3 000000000 ACTIVE YES YES NO 000 255 255 20100114 044504 nbcsapcph016

B 000 001 R/3* 000002454 ACTIVE NO NO NO 004 255 255 20100114 104834 nbcsapcph016

B 000 002 R/3*TIMESTAMP 000002399 DISCONNECTED NO NO NO 004 255 255 20100114 085006 nbcsapcph016

C disconnected from 'BWP', con_hdl=1

B Disconnected from connection con_da={R/3*,2454,1}

C DB2 client driver version '09.01.0001'

C Connected to DB2 server type 'DB2/AIX64'

C Connected to DB2 version '09.01.0001'

C Connect to 'BWP' as 'SAPBWP' schema 'SAPBWP' o.k.; con_hdl=1

B Connection 1 opened (DBSL handle 1)

B Wp Hdl ConName ConId ConState TX PRM RCT TIM MAX OPT Date Time DBHost

B 000 000 R/3 000000000 ACTIVE YES YES NO 000 255 255 20100114 044504 nbcsapcph016

B 000 001 R/3* 000002455 ACTIVE NO NO NO 004 255 255 20100114 104834 nbcsapcph016

B 000 002 R/3*TIMESTAMP 000002399 DISCONNECTED NO NO NO 004 255 255 20100114 085006 nbcsapcph016

C disconnected from 'BWP', con_hdl=1

B Disconnected from connection con_da={R/3*,2455,1}

C

C Thu Jan 14 10:48:35 2010

C DB2 client driver version '09.01.0001'

C Connected to DB2 server type 'DB2/AIX64'

C Connected to DB2 version '09.01.0001'

C Connect to 'BWP' as 'SAPBWP' schema 'SAPBWP' o.k.; con_hdl=1

B Connection 1 opened (DBSL handle 1)

B Wp Hdl ConName ConId ConState TX PRM RCT TIM MAX OPT Date Time DBHost

B 000 000 R/3 000000000 ACTIVE YES YES NO 000 255 255 20100114 044504 nbcsapcph016

B 000 001 R/3* 000002456 ACTIVE NO NO NO 004 255 255 20100114 104835 nbcsapcph016

B 000 002 R/3*TIMESTAMP 000002399 DISCONNECTED NO NO NO 004 255 255 20100114 085006 nbcsapcph016

C disconnected from 'BWP', con_hdl=1

B Disconnected from connection con_da={R/3*,2456,1}

C DB2 client driver version '09.01.0001'

C Connected to DB2 server type 'DB2/AIX64'

C Connected to DB2 version '09.01.0001'

C Connect to 'BWP' as 'SAPBWP' schema 'SAPBWP' o.k.; con_hdl=1

B Connection 1 opened (DBSL handle 1)

B Wp Hdl ConName ConId ConState TX PRM RCT TIM MAX OPT Date Time DBHost

B 000 000 R/3 000000000 ACTIVE YES YES NO 000 255 255 20100114 044504 nbcsapcph016

B 000 001 R/3* 000002457 ACTIVE NO NO NO 004 255 255 20100114 104835 nbcsapcph016

B 000 002 R/3*TIMESTAMP 000002399 DISCONNECTED NO NO NO 004 255 255 20100114 085006 nbcsapcph016

C disconnected from 'BWP', con_hdl=1

B Disconnected from connection con_da={R/3*,2457,1}

C

C Thu Jan 14 10:51:31 2010

C DB2 client driver version '09.01.0001'

C Connected to DB2 server type 'DB2/AIX64'

C Connected to DB2 version '09.01.0001'

C Connect to 'BWP' as 'SAPBWP' schema 'SAPBWP' o.k.; con_hdl=1

B Connection 1 opened (DBSL handle 1)

B Wp Hdl ConName ConId ConState TX PRM RCT TIM MAX OPT Date Time DBHost

B 000 000 R/3 000000000 ACTIVE NO YES NO 000 255 255 20100114 044504 nbcsapcph016

B 000 001 R/3* 000002296 ACTIVE NO NO NO 004 255 255 20100114 105131 nbcsapcph016

B 000 002 R/3*TIMESTAMP 000002399 DISCONNECTED NO NO NO 004 255 255 20100114 085006 nbcsapcph016

];bwpadm@nbcsaprtv016:/usr/sap/BWP/DVEBMGS22/work

former_member286941
Participant
0 Kudos

KEEPFENCED parameter is not set

Data base is up and running at this moement.

bwpadm> db2level

DB21085I Instance "db2bwp" uses "64" bits and DB2 code release "SQL09011" with

level identifier "01020107".

Informational tokens are "DB2 v9.1.0.1", "s061104", "U809676", and Fix Pack

"1".

Product is installed at "/db2/db2bwp/db2_09/v9.1".

Please find the dev_w0 and db2diag's output in next replies

Former Member
0 Kudos

Munish:

At the very beginning you say that your DB version is 8x.

But your db2level output shows version 9x.

What is the DB2 version? Thanks!

former_member286941
Participant
0 Kudos

KEEPFENCED parameter is not set

Data base is up and running at this moement.

bwpadm> db2level

DB21085I Instance "db2bwp" uses "64" bits and DB2 code release "SQL09011" with

level identifier "01020107".

Informational tokens are "DB2 v9.1.0.1", "s061104", "U809676", and Fix Pack

"1".

Product is installed at "/db2/db2bwp/db2_09/v9.1".

Please find the dev_w0 and db2diag's output in next replies

former_member286941
Participant
0 Kudos

KEEPFENCED parameter is not set

Data base is up and running at this moement.

bwpadm> db2level

DB21085I Instance "db2bwp" uses "64" bits and DB2 code release "SQL09011" with

level identifier "01020107".

Informational tokens are "DB2 v9.1.0.1", "s061104", "U809676", and Fix Pack

"1".

Product is installed at "/db2/db2bwp/db2_09/v9.1".

Please find the dev_w0 and db2diag's output in next replies

former_member286941
Participant
0 Kudos

============ PuTTY log 2010.01.14 12:22:28 ============

pwd

/db2/BWP/db2dump

];bwpadm@nbcsaprtv016:/db2/BWP/db2dump

bwpadm> tail -100 db2diag.log

AUTHID : SAPBWP

FUNCTION: DB2 UDB, data management, sqldEndIndexCreate, probe:1

MESSAGE : ADM5542W Index rebuild of index(es) on table "SAPBWP

./BI0/0100000589" completed successfully.

2010-01-14-07.52.14.188706+060 I15188760A379 LEVEL: Severe

PID : 786644 TID : 1 PROC : db2sysc 0

INSTANCE: db2bwp NODE : 000

FUNCTION: DB2 UDB, routine_infrastructure, sqlerReturnFmpToPool, probe:910

MESSAGE : marking fmp as unstable:

DATA #1 : Hexdump, 4 bytes

0x078000000041B660 : 0032 A0B8 .2..

2010-01-14-07.52.14.188903+060 I15189140A2815 LEVEL: Severe

PID : 786644 TID : 1 PROC : db2sysc 0

INSTANCE: db2bwp NODE : 000

FUNCTION: DB2 UDB, routine_infrastructure, sqlerReturnFmpToPool, probe:911

MESSAGE : marking fmp as unstable:

DATA #1 : Hexdump, 488 bytes

0x078000000041B660 : 0032 A0B8 0000 0000 018A 0000 0000 0000 .2..............

0x078000000041B670 : 018B 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041B680 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041B690 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041B6A0 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041B6B0 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041B6C0 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041B6D0 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041B6E0 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041B6F0 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041B700 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041B710 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041B720 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041B730 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041B740 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041B750 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041B760 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041B770 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041B780 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041B790 : 0000 0000 0000 0000 0000 0001 111F 7720 ..............w

0x078000000041B7A0 : 0780 0000 1007 0094 0780 0000 1008 0094 ................

0x078000000041B7B0 : 0001 0000 0000 0000 0000 0000 0000 0513 ................

0x078000000041B7C0 : 0000 05A7 0000 0000 0000 0000 0000 0000 ................

0x078000000041B7D0 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041B7E0 : 0780 0000 0023 AB20 0780 0000 0041 B660 .....#. .....A.`

0x078000000041B7F0 : 0780 0000 002C FF00 0000 0001 0000 0001 .....,..........

0x078000000041B800 : 0000 0000 0000 0000 0000 0001 0000 0000 ................

0x078000000041B810 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

0x078000000041B820 : 0780 0000 001A FB60 0000 0000 0000 0000 .......`........

0x078000000041B830 : 0000 0150 0000 0000 0000 0000 0000 0001 ...P............

0x078000000041B840 : 0100 0000 0000 0000 ........

2010-01-14-07.52.14.189223+060 I15191956A426 LEVEL: Warning

PID : 548998 TID : 1 PROC : db2agent (BWP) 0

INSTANCE: db2bwp NODE : 000 DB : BWP

APPHDL : 0-28 APPID: *LOCAL.db2bwp.100111182348

AUTHID : SAPBWP

MESSAGE : Removing FMP from pool

DATA #1 : Hexdump, 16 bytes

0x0FFFFFFFFFFF0E58 : 0780 0000 0041 B678 0032 A0B8 0000 0000 .....A.x.2......

2010-01-14-07.52.16.548166+060 E15192383A459 LEVEL: Warning

PID : 548998 TID : 1 PROC : db2agent (BWP) 0

INSTANCE: db2bwp NODE : 000 DB : BWP

APPHDL : 0-28 APPID: *LOCAL.db2bwp.100111182348

AUTHID : SAPBWP

FUNCTION: DB2 UDB, data management, sqldBeginIndexCreate, probe:1

MESSAGE : ADM5540W Beginning index rebuild of "1" index(es) on table "SAPBWP

./BI0/0100001133".

2010-01-14-07.52.16.548613+060 E15192843A491 LEVEL: Warning

PID : 548998 TID : 1 PROC : db2agent (BWP) 0

INSTANCE: db2bwp NODE : 000 DB : BWP

APPHDL : 0-28 APPID: *LOCAL.db2bwp.100111182348

AUTHID : SAPBWP

FUNCTION: DB2 UDB, data management, sqldIndexCreate, probe:1

MESSAGE : ADM5541W Rebuilding index with IID "1" in object with ID "225" and

table space ID "33" on table "SAPBWP ./BI0/0100001133".

2010-01-14-07.52.16.572968+060 E15193335A466 LEVEL: Warning

PID : 548998 TID : 1 PROC : db2agent (BWP) 0

INSTANCE: db2bwp NODE : 000 DB : BWP

APPHDL : 0-28 APPID: *LOCAL.db2bwp.100111182348

AUTHID : SAPBWP

FUNCTION: DB2 UDB, data management, sqldEndIndexCreate, probe:1

MESSAGE : ADM5542W Index rebuild of index(es) on table "SAPBWP

./BI0/0100001133" completed successfully.

2010-01-14-07.52.28.758534+060 I15193802A375 LEVEL: Warning

PID : 2486468 TID : 1 PROC : db2logmgr (BWP) 0

INSTANCE: db2bwp NODE : 000

FUNCTION: DB2 UDB, data protection services, sqlpgArchiveLogFile, probe:3180

MESSAGE : Successfully archived log file S0549305.LOG to VENDOR chain 2 from

/db2/BWP/log_dir/NODE0000/.

2010-01-14-07.52.29.763069+060 E15194178A375 LEVEL: Warning

PID : 2486468 TID : 1 PROC : db2logmgr (BWP) 0

INSTANCE: db2bwp NODE : 000

FUNCTION: DB2 UDB, data protection services, sqlpgCleanFailArchPath, probe:4920

DATA #1 : <preformatted>

Unable to find log file 549305 for LOGARCHMETH1. Assume log was archived and continue.

Former Member
0 Kudos

Ok, the sql 1032 means your database manager is not running also db is stopped and 911 is the rollback scenario. Your database stopped like by hand or with a job or an own error. Please provide the output of command db2level, the dev_w0 trace from sap system and also a longer snapshot from db2diag.log.

Please check your dbm cfg parameter KEEPFENCED, probably is set to NO.

Regards

Olaf

Edited by: Olaf Balzer on Jan 14, 2010 8:23 AM

Former Member
0 Kudos

Hi,

it looks like the db connection was lost by sap. Could you please show us the relevant entries in db2diag.log. Are you using a time table in transaction rz04 to switch the number of work processes?

Regards

Olaf