database crashed unexpected

[Former Member]

Sometimes during a normal backup our database (V10.1.0002) goes down and have to be started from clp with crash recovery. The following entries are the first which are written when the error is shown.

2014-04-16-10.30.04.943389+120 E199265E590	LEVEL: Severe
PID	: 28736	TID : 140352753231616 PROC : db2wdog 0 [db2q68]
INSTANCE: db2q68	NODE : 000

HOSTNAME: alihrcs2q

EDUID   : 2

EDUNAME: db2wdog 0 [db2q68]

FUNCTION: DB2 UDB, base sys utilities, sqleWatchDog, probe:20

MESSAGE : ADM0503C  An unexpected internal processing error has occurred. All

	DB2 processes associated with this instance have been shutdown.
	Diagnostic information has been recorded. Contact IBM Support for
	further assistance.

2014-04-16-10.30.05.099303+120 I199856E505	LEVEL: Error
PID	: 28754	TID : 140358231553824 PROC : db2vend (PD Vendor Process - 1)
INSTANCE: db2q68	NODE : 000

HOSTNAME: alihrcs2q

FUNCTION: DB2 UDB, oper system services, DB2VEND_main, probe:1040

MESSAGE : ZRC=0x870F0009=-2029060087=SQLO_EOF "the data does not exist"

DIA8506C Unexpected end of file was reached.

DATA #1 : String, 63 bytes

Fenced vendor process pipe failure, terminating vendor process.

2014-04-16-10.30.06.913208+120 I203841E491           LEVEL: Severe
PID     : 28901                TID : 140220162938624 PROC : db2fmp (
INSTANCE: db2q68               NODE : 000            DB   : Q68    
APPID   : *LOCAL.db2q68.140416083006
HOSTNAME: alihrcs2q
FUNCTION: DB2 UDB, routine_infrastructure, sqlerFmpListener, probe:12
RETCODE : ZRC=0xFFFFFB95=-1131
          SQL1131N  A stored procedure process has been terminated abnormally.
          Routine name: "". Specific name: "".

2014-04-16-10.30.07.137180+120 I204333E485           LEVEL: Error
PID     : 28901                TID : 140220162938624 PROC : db2fmp (
INSTANCE: db2q68               NODE : 000            DB   : Q68    
APPID   : *LOCAL.db2q68.140416083006
HOSTNAME: alihrcs2q
FUNCTION: DB2 UDB, routine_infrastructure, sqlerFmpListener, probe:999
MESSAGE : FMP exiting with error. Last agent registered:
DATA #1 : Hexdump, 4 bytes
0x00007F878D008BD4 : 7800 0000                                  x...

Does anybody here has shown sometimes the same error situation?

Regards

Olaf

[Former Member · Accepted solution]

For your information it was an oom error which kills the most memory consumtion process and this was db2sysc.

Olaf

[Former Member]

After the crash I have a lot of entries in diaglog which looks like the following:

2014-04-22-09.38.21.879789+120 I163455E832           LEVEL: Warning
PID     : 23421                TID : 140025299724032 PROC : db2sysc 0
INSTANCE: db2q68               NODE : 000            DB   : Q68
APPHDL  : 0-11                 APPID: 10.151.17.77.45066.140422073755
AUTHID  : SAPSR3               HOSTNAME: alihrcs2q
EDUID   : 20                   EDUNAME: db2agent (Q68) 0
FUNCTION: DB2 UDB, oper system services, sqloGetEnvInternal, probe:500
MESSAGE : ZRC=0x870F0035=-2029060043=SQLO_NOTAVAIL
          "Cannot honor the conditional request"
          DIA8531C A memory access error has occurred.
DATA #1 : String, 5 bytes
DEFER
DATA #2 : signed integer, 4 bytes
506
DATA #3 : String, 127 bytes
Registry variable cache could not be updated due to a failure upgrading a latch to exclusive mode. Will update cache next time.

2014-04-22-09.38.23.831816+120 E164843E749           LEVEL: Warning
PID     : 23421                TID : 140025152923392 PROC : db2sysc 0
INSTANCE: db2q68               NODE : 000            DB   : Q68
APPHDL  : 0-13                 APPID: 10.151.17.77.45068.140422073755
AUTHID  : SAPSR3               HOSTNAME: alihrcs2q
EDUID   : 55                   EDUNAME: db2agent (Q68) 0
FUNCTION: DB2 UDB, database monitor, sqmSqlTarget::validateTable, probe:60
MESSAGE : ADM2014W  The Event Monitor "SAP_LOCK_EVENTS" detected on table
          "LOCK_PARTICIPANTS" (ID "2") that the size of the column
          "LOCK_OBJECT_TYPE_ID" is smaller than the default size of "2". 
          Therefore, contents will be truncated to the user specified size.

2014-04-22-09.38.23.846066+120 E165593E701           LEVEL: Warning
PID     : 23421                TID : 140025152923392 PROC : db2sysc 0
INSTANCE: db2q68               NODE : 000            DB   : Q68
APPHDL  : 0-13                 APPID: 10.151.17.77.45068.140422073755
AUTHID  : SAPSR3               HOSTNAME: alihrcs2q
EDUID   : 55                   EDUNAME: db2agent (Q68) 0
FUNCTION: DB2 UDB, database monitor, sqmSqlTarget::validateTable, probe:60
DATA #1 : <preformatted>
Event Monitor Name: SAP_LOCK_EVENTS;
TableName: LOCK_PARTICIPANTS;
ColumnName: LOCK_OBJECT_TYPE_ID;
The column size (1) is smaller than the maximum data size (2)
and contents will be truncated to the user specified size!

I cant find any reason for this but its frustrating because this is only shown on qas system not in prod and not in dev.

Regards

Olaf

[hugo_amo]

Hi Olaf,

Was there any FODC created when the db crashed? If so, please attach the stack trace. Else, this would mean (the fact that the FODC was not created) that nothing within the DB caused the DB to crash, therefore I suspect that something outside the database caused the crash.

Regards,

Hugo

[Former Member]

Hi Olaf,

The first problem looks like a system error. I never saw this message before. For solving the error open a message by SAP.

db2diag -rc 0x870F0035

The second one could be solved with evmon_ugrade_tables,

I believe you have tables that didn't match after you migrate the database to higher release.

Best regards,

Joachim

[hugo_amo]

Hi Olaf,

Have you tried disabling db2ats?

Regards,

Hugo

[Former Member]

Hello Hugo,

there are no running admin jobs at the crash time. For this I´ve checked notifier log and events.

Regards

Olaf

[Former Member]

Hi Olaf,

A most likely culprit could be a process which is using or locking a DB2 disk file,

sometimes in Windows an anti-virus program.

Is there anything else written to the db2diag.log or a trace/dump file?

Best regards

Joachim

[Former Member]

Hello Joachim,

nice to see you.

There are no other error entries or something else I´m a little bit confused.

Regards

Olaf

[former_member188883]

Hi Olaf,

It looks like a bug with 10.1 . Can you update the db2 version to 10.1 FP3 and check the results.

FP3 has some similar bug fixes.

Alternatively you may also try

Add CURRENTFUNCTIONPATH='XXX' in the db2cli.ini file where 'XXX' is the schema name

Hope this helps.

Regards,

Deepak Kori

[Former Member]

Hello Deepak,

I´ve updated last week but today the same crash. But only on one database and we have maybe 25 for this customer on the old version 10.1.0002.

Regards

Olaf