Invalid DDIC Objects During System Copy

[Matt_Fraser]

First, a few caveats as I tell you my story of woe and redemption. Some of the techniques I will describe should be used only with extreme care and definite understanding of what you are doing and potential consequences. The story you are about to read involves a sandbox system (just in case), with plenty of backups. I always recommend trying out anything a little dodgy in a sandbox first, of course. Additionally, I should point out that the system involved was on a release that was out of maintenance (even extended maintenance), so obtaining official SAP support for this issue would likely be problematic (not to mention take a lot more time); otherwise I would of course recommend seeking official support. I needed to get this system upgraded, but before I could do that, I needed to get it copied, migrated onto new hardware. Starting with the sandbox.

 

Background

The system involved is an old SRM 5.0 system (out of maintenance) that I am about to upgrade to SRM 7.0 (in maintenance). At the same time, I am migrating the landscape onto new hardware (the old hardware is ten years old), and taking advantage of new database compression capabilities available in SQL Server 2012 (the old system is SQL 2005), as well as redistributing the database to a larger number of disks.

 

In other words, I need to perform a database-independent (R3load-based) homogeneous system copy. This is the ideal way to evenly redistribute the database files, and it achieves the compression in a single step as well.

 

So, I've setup my sandbox environment, but I can't justify production downtime to make a copy for the sandbox. No problem. I start with an ordinary database-dependent (backup/restore) system copy, which involves no downtime on the source system. I just make a copy of the most recent backup, restore that to my sandbox, and run SWPM for the target system installation. Easy-peasy.

 

After ensuring that the sandbox system is running fine post-copy, now I want to export it and re-import it to achieve the disk layout and compression.

 

And that's where my troubles began.

 

The First Export

I prepared my system as usual for the export, de-scheduled all the jobs, shut it down, and fired up SWPM (again, since I had just used it to install this system). The export ran fine, quicker than I expected, but of the total 47 Export Monitor jobs, 1 of them failed. Oh oh. That's not supposed to happen.

 

I drilled into the logs and saw that in one of the export packages there was an "invalid object name" for table /CCM/CUST. This is a table related to the now-defunct module Catalog Content Management, which briefly enjoyed status as the go-to catalog solution after support for Requisite BugsEye dried up and before SRM-MDM launched onto the scene. The only trouble is, we don't use CCM in our system. We briefly experimented with it a number of years ago, but never went anywhere with it. I know for certain that we don't need this table. However, I cannot successfully complete the export with this error.

 

Active DDIC Source Missing or Incorrect

Curious, I go take a look at the original source system, i.e. my production system. In SE16 or SE11 the table doesn't exist. Hmm.

 

SE14 now starts to show the real puzzle. Here the table does seem to exist, or at least SE14 is able to display some data about it. It has a status of Active but also Does not exist in the database.

 

Ok, let's check consistency of database and runtime objects. Sure enough, Table is not created in the database, but I expected that. The runtime object check, however, returns Active DDIC source for /CCM/CUST missing or incorrect. I can display the runtime object, but it's not consistent.

 

I check the object log, and, oops, the log does not lie. The table was deleted, six years ago, via this very utility, SE14, by me.

 

Oh yeah, I kind of remember that now.

 

What Happened

How exactly we got to this point is not really important, but after our abortive attempt nine years ago to install CCM, we simply abandoned it in place. A few years later, however, we upgraded our SRM system from 4.0 to 5.0, and -- digging into the deep memory well here -- we did a "passive deletion" of the CCM component during the upgrade. I don't now recall all the details, but the passive deletion was not entirely successful, and ultimately I finished off deleting the tables involved via SE14. This worked, and the upgrade was a smashing success.

 

Fast forward six years to today, and yes, those tables are gone, but it appears that the SE14 utility did not completely remove the nametab entries for the tables. So, there is no object on the database for them, and there is no active DDIC runtime object, but there is still a data definition, or nametab, for each one.

 

How to Proceed From Here

SE14 is not very useful from this point. As the table doesn't exist, there's nothing for the utility to delete, nor will direct database tools be of any use. The only function active, other than consistency checks, is Create database table, but that's not exactly what I want to do. Still, maybe if I create the table, I can then re-delete it, and this time it will work? Worth a try.

 

But I'm not doing this in production! I'm not even going to do it in DEV. This is what I have a sandbox for, after all.

 

So, I kill the SWPM process and restart the sandbox system. Sure enough, I see the same behavior in SE14 as I saw in DEV and PRD. I create the missing table. It works!

 

Or, does it? Not so fast. It actually doesn't work, because the runtime object is still missing. So now I have a mismatch between database and runtime objects, and I'm no closer to a solution than before. In fact, I'm farther away from one.

 

I delete the table again, and we're back to square one. Despondency sets in. How can I get at the inactive or inconsistent nametab source for a table that doesn't exist in the dictionary?

 

Nametab Adventures

Well, it turns out there's a handy little function module, DD_SHOW_NAMETAB, which indeed shows my rogue ghost table. Great!

 

And, there's another, even handier, function module. Dare I mention it? Yes, it is DD_NAMETAB_DELETE, and it does exactly what it sounds like. I give it a try. I put in my table, /CCM/CUST, and the result seems rather underwhelming. However, now no nametab shows for it, and SE14 now says this table doesn't exist at all. This is promising! Get another backup, shut down the application, and restart SWPM for a new export.

 

More Than One

Failure!

 

Ok, half-failure. The export did not fail on /CCM/CUST this time. Instead, it failed on /CCM/C_ASPT. It looks like there are a bunch of these ghost nametab entries out there. I need to see if I can find them all, because this one-at-a-time business, trying a full export each iteration, will keep me busy until I'm grey(er).

 

SE14 does not find any /CCM tables in a search. The nametabs exist, but apparently not in any search help. No utility seems to be able to successfully search for /CCM/*, nor for that matter for /CCM/C_ASPT, even though I can see /CCM/C_ASPT in SE14 if I enter the name directly. Nor does DD_SHOW_NAMETAB successfully work with a wildcard. It's a one-at-a-time tool.

 

Going back to the logs from my SWPM export, however, I find a file, SAPAPPL1_1.TSK (SAPAPPL1_1.log is where the failure was recorded), which is, as its name implies, something of a task list for that migration monitor package. And, it has a handy list of every table that package was supposed to export, including almost three dozen /CCM tables.

 

Bingo!

 

Success!

Using my list of /CCM tables, I checked each one with SE14 to be certain that there were noexisting database objects and that all of the runtime objects had missing DDIC sources, yet all of the nametabs existed. Once certain that all of them were inconsistent, I ran them all through DD_NAMETAB_DELETE, and, again, restarted my export (third time!).

 

I am happy to report that this resolved the problem, and now you know how I spent my weekend.