NW 7.4 upgrade fails to start shadow instance

[dao_ha]

Hi all,

I'm trying to upgrade our EP (Java only) from NW 7.0 to NW 7.4. We did a test on a POC server last Summer and it went fine.

Now, it's time to do real upgrade on our servers; of course, the old SUM version doesn't work anymore. I used MOPZ to generate new stack files, download the latest components and all.

Somehow, the SUM kept failing at step 5.5 - Preprocessing when it tries to Start shadow instance SCS.

The log only indicates that SUM tried to start the instance a 20 times (with 15-second pauses in between) so it gave up. The last part of the log shows

------------------------

Feb 11, 2014 1:50:36 PM [Info  ]: All retrieved instances : 3

Feb 11, 2014 1:50:36 PM [Info  ]: Correctly detected instance: 70 on host mydev

Feb 11, 2014 1:50:36 PM [Info  ]: Correctly detected instance: 2 on host mydev

Feb 11, 2014 1:50:36 PM [Info  ]: Correctly detected instance: 0 on host mydev

Feb 11, 2014 1:50:36 PM [Info  ]: Instance has status STOPPED.

Feb 11, 2014 1:50:36 PM [Info  ]: Starting instance number 2 on host mydev with timeout of 7200 seconds.

Feb 11, 2014 1:50:36 PM [Error ]: The maximum retry count (20) was reached.

Feb 11, 2014 1:50:36 PM [Error ]: The following problem has occurred during step execution: com.sap.sdt.util.diag.DiagException: Could not start SAP instance  with number 2.

Could not send the command to start the instance with number 2 on host mydev.

Error starting instance 2 on host mydev with timeout of 7200 seconds.

Errors while starting the instance.

javax.xml.ws.soap.SOAPFaultException: Invalid Credentials

--------------------------

The only SAP Note I found is 1937296 where it says you'd need the latest kernel. But I've already had the latest kernel.

What bothers me is "Invalid Credentials" --> which credentials?

I tried several different things. Reset the upgrade, cleaned up the SUM directory, re-start the upgrade from scratch several times. I've also tried to re-start the instance; even re-started the server a couple of times ... to no avail 😞

I also tried to run the sapstartsrv command line

## ../SUM/sdt/data/kernel/sapstartsrv pf=../../DPK_SCS02_mydev -D

and I'd get the error

--------------------

Could not open the ICU common library.

   The following files must be in the path described by

   the environment variable "LD_LIBRARY_PATH":

   libicuuc.so.50, libicudata.so.50, libicui18n.so.50 [/bas/741_REL/src/flat/nlsui0.c 1535] pid = 18978

LD_LIBRARY_PATH is currently set to /usr/sap/DPK/SYS/exe/run:/oracle/client/11x_64/instantclient  [/bas/741_REL/src/flat/nlsui0.c 1538] pid = 18978

--------------------

Any help and/or suggestion would be greatly appreciated.

Regards,

Dao

[Former Member · Accepted solution]

Hi

Did you check note 1563660 ? I faced the same issue and by going through this below steps I was able to get through the error.

Stop all the SAP process and setup below profile parameter

Set profile parameter

service/protectedwebmethods = SDEFAULT

Once you set it check the

<exe direcotry> sapcontrol -nr <instance numebr> -function GetStartProfile | grep protectedwebmethods

if your instance is 00 then

sapcontrol -nr 00 -function GetStartProfile | grep protectedwebmethods

service/protectedwebmethods = SDEFAULT

Start SAP using sapcontrol

./sapcontrol -nr 00 -function Start

sapstartsrv pf= instanc profile -D

Start Shadow using sapcontrol which is in the shadow exe directory

./sapcontrol -nr <shadow isntance number> -function Start

Start sapstartsrv process

sapstartsrv pf=  shadow instanc profile -D

Also if you are on unix o/s you have to check the  sapuxuserchk permissions.

This should fix the issue

[dao_ha · Accepted solution]

Hi Chandra,

Thanks for your help. I checked the note and tried with all 3 options SDEFAULT, DEFAULT, NONE.

Unfortunately, it didn't resolve the issue.

SAP just got back to me and asked me to try SAP Note 1737528. I'll try do that tomorrow and see how it goes.

Thanks again.

Dao

[dao_ha · Accepted solution]

I applied the Note 1737528 as suggested by SAP, re-tried the SUM from scratch and ... got the same problem.

We'll be trying something else.

Dao

[dao_ha · Accepted solution]

Hi all,

With the help of our great Basis member, we looked at SAP Note 1563660 and found out that there is another sapuxuserchk under the SUM directory that I overlooked.

We changed the ownership/permissions for that file and were able to continue.

Thanks everyone for helping out.

Regards,

Dao

[Former Member · Accepted solution]

We also faced similar issue...Resolved with the help of SAP Note  1563660 - sapcontrol, <sid>adm authorization issues (SUM)…

Regards

Maruthi V

[former_member114335]

Dao - we had a number of issues when upgrading production.  It finally appeared to be a problem related to having 6 servers in the production landscape.  We removed 4 servers for upgrade and upgraded with only 2 and it finally worked.

[dao_ha]

Hi Eric,

Thanks for sharing your experience. We only have 2 servers in our landscape and it doesn't seem to be a problem.

Regards,

Dao

[dao_ha]

Thanks, guys, for looking into this.

@Nicholas: Our portal is Java only so there's no ABAP stack involved. I did check the SMP and we do have the latest kernel version (12/2013) downloaded.

@Jamil: I looked at the note. It seems to apply to NW 7.3 or 7.2 only. It also suggests that we install the latest kernel. I'll dig a bit deeper on this note to see if there's something I can use.

@RB: I don't see much in the logs. Where can I find the work process trace of the shadow system?

Regards,

Dao

[Reagan]

Hello Dao

Please check the logs at:

/<upgrade_directory>/abap/SID/DVEBMGSXX/work

Regards

RB

[dao_ha]

Hi RB,

Thanks for your help. Mine is a Java-only upgrade so there's no ABAP directories under the SUM directory 🙂

Thanks,

Dao

[Reagan]

If this is a Java stack then there should be a java sub directory. So replace abap with java and check the logs.

/<upgrade_directory>/java/SID/DVEBMGSXX/work

Regards

RB

[dao_ha]

Hi RB,

There's no directory

/<upgrade_directory>/java/SID/...

If you meant the sapstartsrv.log, it's under

/<upgrade_directory>/sdt/SID/SCSxx/work/

and all it only has

---------------------------------------------------
trc file: "sapstartsrv.log", trc level: 0, release: "741"
---------------------------------------------------
pid        8342

[Thr 139748351395680] Wed Feb 12 11:55:03 2014
No halib defined => HA support disabled
CCMS agent initialization for instance type SCS: return code 0.
CCMS agent start: return code 0.
Initializing SAPControl Webservice
Starting AutoRestart thread
SapSSLInit failed => https support disabled
Starting WebService thread
AutoRestart thread started
Webservice thread started, listening on port 50313
Trusted http connect via Unix domain socket '/tmp/.sapstream50313' enabled.
[Thr 139748349998848] Wed Feb 12 11:55:04 2014
Auto PSE update thread started

---------------------------------------------------

Regards,

Dao

[Reagan]

Hello Dao

If possible stop the SAP shadow and main system

Clean the shared memory segments from the OS level.

Delete all the stream files under /tmp

Start SAP and see if this helps.

Regards

RB

[dao_ha]

Hi RB,

I tried as suggested. Unfortunately, it didn't help 😞

Regards,

Dao

[Reagan]

In that case the only option is to raise an OSS message with SAP.

Follow this SAP note

1019929 - SAP Java Upgrade Problems

Regards

RB

[dao_ha]

Hi RB,

Thanks for your help.

I raised an OSS message with SAP 2 days ago and am still waiting for their first response 🙂

Regards,

Dao

[deeraj_shetty]

Hi Dao,

Did your issue resolve?

Regards,

Deeraj Shetty

[dao_ha]

Hi Deeraj,

Unfortunately, not yet. I'm still waiting for SAP. Things don't move very fast.

Do you have any suggestions?

Thanks,

Dao

[deeraj_shetty]

Dear Dao,

I had faced similar situation before and fixed it by following below steps:

1. sapcontrol -nr <Shadow instance> -function Stop

2, sapcontrol -nr <Shadow instance> -function GetProcessList

3. Kill junk sapstartsrv process for shadow instance

4. sapcontrol -nr <Shadow instance> -function Start

5. If this doesn’t’ work, then stop sum, stop SAP, clean ipc and  then restart SUM again

Hope it helps in fixing your issue.

Regards,

Deeraj Shetty

[dao_ha]

Thanks, Deeraj.

I tried as suggested ... the issue is still there.

Dao

[Reagan]

Hello

What does the work process trace of the shadow system say ?

Regards

RB

[former_member182034]

hi Dao,

did you check this note If not then please check it.

1590392 - javax.xml.ws.soap.SOAPFaultException guring update

Regards,

Jamil

[nicholas_chang]

Hi Dao,

Could you please check the relevant trace/logs file in /sum/abap/SID/DVE*/work and try to see any clue.

Besides, do ensure you download the latest kernel from SMP. Dont rely on the kernel downloaded from MOPZ, as normally the kernel would not be the latest.

If there's newer or later Kernel and updated R3* in SMP, download them and extract it to \SUM\abap\exe. Make a backup or \exe prior to that.

Lastly, you can try to start shadow instance manually with below command:

cd to SUM\abap\bin

./SAPup startshd

Hope it helps,

Cheers,

Nicholas Chang