SAP is running with corrupted DB6 Database with many "Bad Page Header" . .

[former_member524429]

Hello Friends,

One of our DB6 Database is showing multiple entries in db2diag.log file, while performing SAP Local client copy, SGEN, SAP System Copy, REORGCHK for All, RUNSTAT for single Table......

RETCODE : ZRC=0x87020036=-2029912010=SQLB_BADHDR "Bad Page Header"
                   DIA8547C An error occurred in a database page header.

System Information:

SAP system : Solution Manager 70 , Support Stack 15

Database : DB2 v9.1.700.855 , FP7

Operating system: Windows Server 2003 Ent. x64 Edition

Currently, All things are working fine with the current status of DB6 Database. SAP Solman system is starting and shutting down in normal way whenever it is required. In DB6COCKPIT, not even single alert is there for inconsistent objects, and for other factors. Even, At present, I am able to perform both compressed /uncompressed Online/Offline backup of the System Database.

Previously, the system was maintained by other BASIS administrator. The problem came to know by me, when i ran SGEN after successfully applying ST-A/PI add-on in system, as per as EarlyWatch report notification. The corruption might happened before 6 to 7 months ago, as i have seen the same messages in db2diag.log files, when the same problem was identified by me recently. And moreover, as per as our Backup cycle, all dedicated Backup Tapes are overwritten with existing status of SAP Solman Database within this 6 to 7 months long period, so I can not restore the consistent Database backup to correct the problem.

The ABAP dumps which came as a result of SGEN and of some operations are as followed:

DBIF_REPO_INTERNAL_ERROR

DBIF_REPO_SQL_ERROR (SQL error "-1034" occurred when accessing program, SQL error "-99999" occurred when accessing program)

GEN_BRANCHOFFSET_LIMIT_REACHED

etc...etc...

I tried to diagnosis the cause of this problem using DB2DART tool and done the analysis of all tablespaces. Out of 37 Tablespaces, multiple Tablespaces are found with error/corruptions.

Due to the corruption in multiple Database Obejcts, I am not able to do SAP Local client copy, SGEN, SAP System Copy, REORGCHK for All, RUNSTAT for single Table... and other Database related operations.

Is there any alternative to correct this corrupted database excep Backup/Restore database?

I am looking for your cooperations to solve this unexpected critical Problem.

Regards,

Bhavik G. Shroff

Edited by: Bhavik G. Shroff on Nov 12, 2009 11:31 AM

[paul_power]

Hello Bhavik,

If the issue occured on an index object there would be the possibility to simply mark the index as bad and recreate it (this can be done via dbdart tool) Unfortunately, for the entry Pagekey: PPNum:5409475

If the type is 0 then it is a table. If it is 1 then it is an index.

Here we see that the object in tablespace 59, obkect 9 is a data object. It is not possible to repair a data object (you could drop the table and recreate it but this would of course lead to data loss)

When a data type object is involved in a bad data page error, the correct solution would be to restore from a consistent backup.

[Former Member]

Bhavik,

Are these bad pages in the table "data" pages or are they present in the "index" pages ? db2dart will be able to tell you that. You can also pick up the page number and offset from the db2diag.log and dump that page to see if its zeroed out or corrupted. If the "indexes" are corrupted, you might be able to rebuild indexes for these tables and get around the problem without doing a restore. We had a similar situation where we could get around by rebuilding the indexes.

- Sameer

[Former Member]

Could you kindly tell how to do following?

You can also pick up the page number and offset from the db2diag.log and dump that page to see if its zeroed out or corrupted.

[Former Member]

Ashley,

Fortunately the db2diag.log provides a lot of good debugging information. Here is an excerpt of the information that we captured for a (different) problem we were having:

2009-03-26-17.22.23.655840000 I6020445A3130 LEVEL: Severe

PID : 1314904 TID : 5913 PROC : db2sysc 0

INSTANCE: db2bq2 NODE : 000

EDUID : 5913 EDUNAME: db2pclnr (BQ2) 0

...........

0x0000000110692608 : 0000 0000 0000 0000 FFFF FFFF FFFF FFFF ................

0x0000000110692618 : 0000 0002 850F 000C ........

DATA #3 : Buffer page descriptor, PD_TYPE_SQLB_BPD, 152 bytes

Pagekey: PPNum:5409475

objectPageNum: 153219

bucketGroupHashIndex: 1817

hashnext: 0x0000000000000000

hashprev: 0x0700000063240858

osslatch:

+

Basically the information you are looking for is the PageKey which tells you that the problem happened in tablespace id 59, tableid 9 pagenumer 5409475. You can then use db2dart to dump that particular page and examine the contents (in our case it was zeroed out due a filesystem problem).

db2dart <db> /DP /TSI 59 /PS 5409475 /NP 0

There are various other dump options that you can use. I have found this to be very useful in diagnosing these kind of problems.

- Sameer

[former_member524429]

Dear Sameer,

Sorry for the delay in response. I was busy in other critical activities.

Anyway, thanks for your valuable reply with such kind of good problem example. And also Thanks to Ashley for asking such a useful question.

I am having some average knowledge in DB2, as I am working with DB2 since last 2 months.

As described in my problem description, I have done lot of things using DB2DART utilities as per suggested by SAP Support team before some days. And at the end , they called for re-building the SAP Solman or do the restore from Consistent DB Backup. But, If is possible, i want to correct the things without going to try for that last option.

I am not getting the starting point from where I should have to start the correcting procedure using available tools.

Shall I have any steps guide, so that I can find the things step by step and do the required analysis to correct the affected Objects ?

Any Alternatives are there to repair such kind of Database without going for Last option (Re-building or Restoration) ?

Regards,

Bhavik G. Shroff