Dear Experts,
We have a problem with our standby HADR server with the following error message:
2014-07-24-16.48.46.985569+420 E133880710A860 LEVEL: Error
PID : 20578344 TID : 15422 PROC : db2sysc 0
INSTANCE: db2p01 NODE : 000 DB : P01
APPHDL : 0-3936 APPID: *LOCAL.DB2.140724094804
EDUID : 15422 EDUNAME: db2redom (P01) 0
FUNCTION: DB2 UDB, buffer pool services, sqlbAlterPool, probe:155
MESSAGE : ADM6083E An error occurred while redoing an alter tablespace
operation against table space "ZSAPBTAB26I" (ID "79") This error will
be temporarily ignored while the remainder of the transaction is
replayed. If the alter operation is eventually rolled back then the
error will be discarded. However, if the operation is committed then
this error will be returned, stopping recovery against the table
space.
2014-07-24-16.48.46.986007+420 I133881571A470 LEVEL: Error
PID : 20578344 TID : 15422 PROC : db2sysc 0
INSTANCE: db2p01 NODE : 000 DB : P01
APPHDL : 0-3936 APPID: *LOCAL.DB2.140724094804
EDUID : 15422 EDUNAME: db2redom (P01) 0
FUNCTION: DB2 UDB, buffer pool services, sqlbAlterPool, probe:135
MESSAGE : ZRC=0x850F000C=-2062614516=SQLO_DISK "Disk full."
DIA8312C Disk was full.
2014-07-24-16.48.46.986276+420 E133882042A652 LEVEL: Error
PID : 20578344 TID : 15422 PROC : db2sysc 0
INSTANCE: db2p01 NODE : 000 DB : P01
APPHDL : 0-3936 APPID: *LOCAL.DB2.140724094804
EDUID : 15422 EDUNAME: db2redom (P01) 0
FUNCTION: DB2 UDB, buffer pool services, sqlbAlterPoolAct, probe:10
MESSAGE : ADM6084E An attempt is being made to commit an alter operation
against table space "ZSAPBTAB26I" (ID "79") but a previous error is
preventing this from being done. Resolve the original error before
attempting the recovery again.
2014-07-24-16.48.46.986690+420 I133882695A472 LEVEL: Error
PID : 20578344 TID : 15422 PROC : db2sysc 0
INSTANCE: db2p01 NODE : 000 DB : P01
APPHDL : 0-3936 APPID: *LOCAL.DB2.140724094804
EDUID : 15422 EDUNAME: db2redom (P01) 0
FUNCTION: DB2 UDB, buffer pool services, sqlbAlterPoolAct, probe:10
MESSAGE : ZRC=0x850F000C=-2062614516=SQLO_DISK "Disk full."
DIA8312C Disk was full.
2014-07-24-16.48.46.986911+420 I133883168A531 LEVEL: Severe
PID : 20578344 TID : 15422 PROC : db2sysc 0
INSTANCE: db2p01 NODE : 000 DB : P01
APPHDL : 0-3936 APPID: *LOCAL.DB2.140724094804
EDUID : 15422 EDUNAME: db2redom (P01) 0
FUNCTION: DB2 UDB, data management, storMgrAction, probe:298
MESSAGE : Error during str mgr pending action. Trace data:
DATA #1 : Hexdump, 8 bytes
0x07000000013FA378 : 850F 000C 0000 004F .......O
2014-07-24-16.48.46.987075+420 I133883700A612 LEVEL: Severe
PID : 20578344 TID : 15422 PROC : db2sysc 0
INSTANCE: db2p01 NODE : 000 DB : P01
APPHDL : 0-3936 APPID: *LOCAL.DB2.140724094804
EDUID : 15422 EDUNAME: db2redom (P01) 0
FUNCTION: DB2 UDB, data management, sqldmpnd, probe:189
MESSAGE : Error executing pending list log record. Trace data:
DATA #1 : Hexdump, 24 bytes
0x07000000013FA498 : 850F 000C 0000 0DAC 0000 0004 F28B 2A80 ..............*.
0x07000000013FA4A8 : 0000 1808 2AB3 5552 ....*.UR
2014-07-24-16.48.46.987240+420 I133884313A1346 LEVEL: Severe
PID : 20578344 TID : 15422 PROC : db2sysc 0
INSTANCE: db2p01 NODE : 000 DB : P01
APPHDL : 0-3936 APPID: *LOCAL.DB2.140724094804
EDUID : 15422 EDUNAME: db2redom (P01) 0
FUNCTION: DB2 UDB, data management, sqldCriticalSectionEnd, probe:9298
CALLED : DB2 UDB, data management, sqldmpnd
RETCODE : ZRC=0x850F000C=-2062614516=SQLO_DISK "Disk full."
DIA8312C Disk was full.
CALLSTCK:
[0] 0x09000000053510D0 pdLogRC + 0xF8
[1] 0x09000000061CBAC8 sqldmpnd__FP8sqeAgentiPcP9SQLP_LSN8P15SQLD_RECOV_INFO + 0x440
[2] 0x09000000064CE6E8 sqlptppl__FP8sqeAgent + 0xA04
[3] 0x0900000006175BA0 sqlpRFWppl__FP8sqeAgentP9SQLP_DBCBP11SQLP_TENTRYP10SQLP_FRAPPP10
SQLP_FRSCBP9SQLP_LSN8Ui + 0x78
[4] 0x0900000006321E60 sqlpRecDbRedo__FP8sqeAgentP8SQLP_ACBP9SQLP_DBCBP10SQLP_FRAPPP11S
QLP_TENTRYP16SQLPR_LOGREC_DISP10REDO_INPUT + 0x1B8
[5] 0x090000000619CC08 sqlprProcDPSrec__FP10SQLPR_PRCBUiP20SQLPR_PR_QUEUE_ENTRY + 0x40
[6] 0x090000000618E870 sqlpPRecReadLog__FP8sqeAgentP8SQLP_ACBP9SQLP_DBCB + 0x17F0
[7] 0x09000000062EA454 sqlpParallelRecovery__FP8sqeAgentP5sqlca + 0x14
[8] 0x0900000005BB0834 sqleSubCoordProcessRequest__FP8sqeAgent + 0x210
[9] 0x090000000534AFA4 RunEDU__8sqeAgentFv + 0x3C0
2014-07-24-16.48.47.641278+420 E133885660A922 LEVEL: Critical
PID : 20578344 TID : 15422 PROC : db2sysc 0
INSTANCE: db2p01 NODE : 000 DB : P01
APPHDL : 0-3936 APPID: *LOCAL.DB2.140724094804
EDUID : 15422 EDUNAME: db2redom (P01) 0
FUNCTION: DB2 UDB, base sys utilities, sqeLocalDatabase::MarkDBBad, probe:10
MESSAGE : ADM14001C An unexpected and critical error has occurred:
"DBMarkedBad". The instance may have been shutdown as a result.
"Automatic" FODC (First Occurrence Data Capture) has been invoked and
diagnostic information has been recorded in directory
"/db2/P01/db2dump/FODC_DBMarkedBad_2014-07-24-16.48.46.987605/".
Please look in this directory for detailed evidence about what
happened and contact IBM support if necessary to diagnose the
problem.
2014-07-24-16.48.47.641797+420 E133886583A431 LEVEL: Severe
PID : 20578344 TID : 15422 PROC : db2sysc 0
INSTANCE: db2p01 NODE : 000 DB : P01
APPHDL : 0-3936 APPID: *LOCAL.DB2.140724094804
EDUID : 15422 EDUNAME: db2redom (P01) 0
FUNCTION: DB2 UDB, base sys utilities, sqeLocalDatabase::MarkDBBad, probe:10
MESSAGE : ADM7518C "P01 " marked bad.
2014-07-24-16.48.47.642192+420 I133887015A444 LEVEL: Severe
PID : 20578344 TID : 15422 PROC : db2sysc 0
INSTANCE: db2p01 NODE : 000 DB : P01
APPHDL : 0-3936 APPID: *LOCAL.DB2.140724094804
EDUID : 15422 EDUNAME: db2redom (P01) 0
FUNCTION: DB2 UDB, base sys utilities, sqeLocalDatabase::MarkDBBad, probe:210
MESSAGE : Database logging stopped due to mark db bad.
2014-07-24-16.48.47.654457+420 I133887460A462 LEVEL: Severe
PID : 20578344 TID : 15422 PROC : db2sysc 0
INSTANCE: db2p01 NODE : 000 DB : P01
APPHDL : 0-3936 APPID: *LOCAL.DB2.140724094804
EDUID : 15422 EDUNAME: db2redom (P01) 0
FUNCTION: DB2 UDB, relation data serv, sqlrr_signal_handler, probe:100
MESSAGE : DIA0505I Execution of a component signal handling function has begun.
2014-07-24-16.48.47.654703+420 I133887923A599 LEVEL: Severe
PID : 20578344 TID : 15422 PROC : db2sysc 0
INSTANCE: db2p01 NODE : 000 DB : P01
APPHDL : 0-3936 APPID: *LOCAL.DB2.140724094804
EDUID : 15422 EDUNAME: db2redom (P01) 0
FUNCTION: DB2 UDB, relation data serv, sqlrr_signal_handler, probe:125
MESSAGE : MASTER TIME INFO
DATA #1 : Hexdump, 32 bytes
0x07000006EDBA97D8 : 0000 0000 53D0 D654 027B 0000 0009 B26B ....S..T.{.....k
0x07000006EDBA97E8 : 001C C749 E8AA 010B 0000 0000 001F 3A61 ...I..........:a
2014-07-24-16.48.47.656345+420 I133888523A552 LEVEL: Severe
PID : 20578344 TID : 15422 PROC : db2sysc 0
INSTANCE: db2p01 NODE : 000 DB : P01
APPHDL : 0-3936 APPID: *LOCAL.DB2.140724094804
EDUID : 15422 EDUNAME: db2redom (P01) 0
FUNCTION: DB2 UDB, access plan manager, sqlra_sqlW_raw_dump, probe:150
MESSAGE : Workspace info dumped! Check
/db2/P01/db2dump/FODC_DBMarkedBad_2014-07-24-16.48.46.987605/20578344.
15422.000.apm.bin for additional data.
2014-07-24-16.48.47.656540+420 I133889076A599 LEVEL: Severe
PID : 20578344 TID : 15422 PROC : db2sysc 0
INSTANCE: db2p01 NODE : 000 DB : P01
APPHDL : 0-3936 APPID: *LOCAL.DB2.140724094804
EDUID : 15422 EDUNAME: db2redom (P01) 0
FUNCTION: DB2 UDB, access plan manager, sqlra_sqlW_raw_dump, probe:175
MESSAGE : MASTER TIME INFO
DATA #1 : Hexdump, 32 bytes
0x07000006EDBA97D8 : 0000 0000 53D0 D654 027B 0000 0009 B26B ....S..T.{.....k
0x07000006EDBA97E8 : 001C C749 E8AA 010B 0000 0000 001F 3A61 ...I..........:a
2014-07-24-16.48.47.657079+420 I133889676A557 LEVEL: Severe
PID : 20578344 TID : 15422 PROC : db2sysc 0
INSTANCE: db2p01 NODE : 000 DB : P01
APPHDL : 0-3936 APPID: *LOCAL.DB2.140724094804
EDUID : 15422 EDUNAME: db2redom (P01) 0
FUNCTION: DB2 UDB, access plan manager, sqlra_cache_full_dump, probe:25
MESSAGE : Package Cache info dumped! Check
/db2/P01/db2dump/FODC_DBMarkedBad_2014-07-24-16.48.46.987605/20578344.
15422.000.apm.bin for additional data.
2014-07-24-16.48.47.658013+420 I133890234A475 LEVEL: Severe
PID : 20578344 TID : 15422 PROC : db2sysc 0
INSTANCE: db2p01 NODE : 000 DB : P01
APPHDL : 0-3936 APPID: *LOCAL.DB2.140724094804
EDUID : 15422 EDUNAME: db2redom (P01) 0
FUNCTION: DB2 UDB, relation data serv, sqlrr_signal_handler, probe:350
MESSAGE : DIA0506I Execution of a component signal handling function is
complete.
2014-07-24-16.48.47.659089+420 I133890710A574 LEVEL: Event
PID : 20578344 TID : 15422 PROC : db2sysc 0
INSTANCE: db2p01 NODE : 000 DB : P01
APPHDL : 0-3936 APPID: *LOCAL.DB2.140724094804
EDUID : 15422 EDUNAME: db2redom (P01) 0
FUNCTION: DB2 UDB, base sys utilities, sqeLocalDatabase::MarkDBBad, probe:10
STOP : AUTOMATIC FODC : DBMarkedBad : success
IMPACT : None
DATA #1 : String, 95 bytes
Check /db2/P01/db2dump/FODC_DBMarkedBad_2014-07-24-16.48.46.987605/ for diagnostic inform
ation.
2014-07-24-16.48.47.659327+420 E133891285A523 LEVEL: Error
PID : 20578344 TID : 15422 PROC : db2sysc 0
INSTANCE: db2p01 NODE : 000 DB : P01
APPHDL : 0-3936 APPID: *LOCAL.DB2.140724094804
EDUID : 15422 EDUNAME: db2redom (P01) 0
FUNCTION: DB2 UDB, data protection services, sqlptppl, probe:2220
MESSAGE : ZRC=0x87040055=-2029780907=SQLD_PRGERR "Unknown PROGRAM ERROR"
DIA8576C A data management services programming error occurred.
2014-07-24-16.48.47.659558+420 I133891809A520 LEVEL: Error
PID : 20578344 TID : 15422 PROC : db2sysc 0
INSTANCE: db2p01 NODE : 000 DB : P01
APPHDL : 0-3936 APPID: *LOCAL.DB2.140724094804
EDUID : 15422 EDUNAME: db2redom (P01) 0
FUNCTION: DB2 UDB, recovery manager, sqlpRecDbRedo, probe:2130
RETCODE : ZRC=0x87040055=-2029780907=SQLD_PRGERR "Unknown PROGRAM ERROR"
DIA8576C A data management services programming error occurred.
It occured when enlarging tablespace by adding container however standby server does not have enough space to do it.
My question is:
is there any workaround rather than backup restore?
Thank you in advance.
Regards,
Rudi
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.