Netweaver 7.4 SR2 Install Stop Error - startJava was executed with status ERROR (err code: FCO-00011)

[Former Member]

Hi All,

I have been installing Netweaver 7.4 SR2 for Java on Windows 8.1 Enterprise SAP - Image Version: 2014.06 64-bit 16GB ram, using MSSQL, the install runs happily up to the step where it tries to start the instance, however it fails after trying to start for 5 minutes.

Looking through the error logs on the log analyser the error I have found is:

FCO-00011

ERROR      2015-04-30 12:29:41.93 [csistepexecute.cpp:1158]

FCO-00011  The step startJava with step key NW_CI_Instance_StartJava -> startJava  was executed with status ERROR

ERROR 2015-04-30 12:29:41.93 [csistepexecute.cpp:1158]
FCO-00011 The step startJava with step key |NW_Java_OneHost|ind|ind|ind|ind|0|0|NW_Onehost_System|ind|ind|ind|ind|onehost|0|NW_CI_Instance|ind|ind|ind|ind|ci|0|NW_CI_Instance_StartJava|ind|ind|ind|ind|17|0|startJava was executed with status ERROR ( Last error reported by the step: Java process server0 of instance A01/J00 reached state Stopped after having state Waiting to start. Giving up.).

I have tried all the usual things like restarting the install, trying a completely fresh install, starting the instance manually but it will always fail at this point.

Has anyone else encountered this error, or can suggest further logs to inspect?

Many thanks in advance!

Hanif

[Former Member · Accepted solution]

I finally got to the bottom of the problem.   Thanks to some guidance from an internal incident ticket I filed with the BC-JVM / BC-JAS-COR component in support.wdf.sap.corp, they noticed that the exception from the stacktrace was java.io.IOException: Unable to establish loopback connection which in turn was caused by a socket connection exception: java.net.SocketException: Address family not supported by protocol family: bind.    The support individual mentioned that NW only supports the IPv4 address protocol (even though there are SAP notes I saw that say otherwise).    Anyhow, I checked my ipconfig, and as you can see my ethernet connection was using IPv4 but below in the highlighted area, something was occupying the IPv6 address via the tunnel adapter:

Upon closer inspection, I noticed I had an NFS drive attached that was created when I used the sapmnt daemon to mount a media drive to the servers in wdf.  Once I disconnected the drive, I restarted my machine and ran ipconfig again and that tunnel adapter entry was no longer populated and all was left was the IPv4 address being used by my ethernet adapter.

I resumed the installation and when it got to the java start step (where it was previously failing), I started to see angels descend from the heavens and I began to hear soft violins in the background.   Once the angels had descended they all gave me a gracious smile and the installation completed without incident at which point the angels along with the violin music ascended back up to heaven.

Thanks so much to all that chimed in and helped me!  Truly appreciated!

Hanif

[Matt_Fraser · Accepted solution]

Hanif,

Not only is that a thorough description of a fairly obscure technical issue and its resolution (thank you for that, btw), but it was fun to read, too. Soft violins indeed!

Cheers,

Matt

[Former Member]

Update:

Yesterday my windows 8.1 machine was refreshed and restarted by required windows updates (forced through sap software corner).    And the same issue manifested itself.   ipconfig this time did not show the additional tunnel adapter entry.    Ultimately, I had to disable ipv6 altogether to get it to work (via official microsoft 'fix this' which supposedly 'correctly' sets the requisite registry values using this link: https://support.microsoft.com/en-ca/kb/929852 ).    Some facet of the windows 8 update dealing with the ipv6 protocol does not play nice with NW.    Although they recommend not to disable ipv6 altogether as it may affect other apps, this was the only solution on hand that worked for NW.    The battle ensues.....

[Former Member]

Hi Hanif,

Thanks for your resolution methods: I faced exactly the same issue. May I add that it is necessary to completely disable ipv6 completely. Disabling ipv6 for tunnel and non-tunnel does not suffice.

Kind regards,

Gerwin

[Former Member]

Yes I found that to be the case as well Gerwin.   Thanks for chiming in and glad you got your issues resolved hopefully quicker than me! 

[Sriram2009]

Hi Hanif

I think you can do the full system restart (Stop the SAP installation and then restart the windows system) and then start the installation on same phase

BR

SS

[Former Member]

Hi Sriram, I have tried halting the install, re-booting and trying again.   I've also tried halting the install, trying to start the instance manually.   All to no avail

[hemanth2]

HI Hanif,

Please do the below:

cleanipc <SCS-instance-no> remove

(where <SCS-instance-no> is the instance number of the SCS).

Then execute the 'cleanipc <nr> remove' with the root user to ensure that none of the SHM segments are present.  Once this is done, stop the complete SAP server.Now start it again (first start the SCS<nr> and then the server nodes).If the issue persists, attach the whole work folder.

regards,
Hemanth
SAP AGS

[Former Member]

Hi Hemanth, thanks for your reply.....in googling I see that cleanipc is used for removing shared memory.  Given that I've restarted my machine a few times would that still apply?   Do you think that has something to do with the cluster manager and the join port issue?    In any event, I'm willing to try anything!   Would you be able to be a bit more explicit in the instructions?   Where would I find the cleanipc executable on windows?   Sorry for the noob question, but when you say SCS-instance-no what would that be?  Here is a snapshot of my sapmmc:

Also, I'm unsure what the acronyms 'SHM' and 'nr' means.... (too many to keep track of in the sap world)

thanks again,

Hanif

[Matt_Fraser]

If I'm remembering my SAPMMC icons correctly, then your SCS instance number would be 01 from that picture.

[Matt_Fraser]

Hi Hanif,

A couple of things to check right off. Are you sure that this installation is supported on Windows 8.1? The desktop Windows OS versions are not listed in the PAM; only the Server OS versions.

Next: what version of SQL Server are you using? Starting with SP08, which is what NetWeaver 7.4 SR2 includes, a minimum of SQL Server 2012 is required. Older support stack levels of 7.4 could work on older releases of SQL, but not SR2.

Then, next, which version of the kernel are you including in the installation? SR2/SP8 requires the 742 kernel. It won't work with 740 or 741.

Are you using the latest and greatest patch level of SWPM?

Cheers,

Matt

[Former Member]

Hi Matt, first off thank you for your response.  

I have colleagues who have successfully installed this version on win 8.1 desktop - moreover, I'm limited to what is provided to us on our laptops which is this OS version.

I am using SQL Server 2012 and for the kernel I'm using the 742 kernel (LDC_742_KRN_WINX86_64 referenced from the MID.XML file)

As for the swpm, not sure what the patch level is as I'm using it from a folder that was downloaded for us by our systems group along with all of the installation image files for NW & MSSQL.    Note that this installation has worked for my fellow colleagues some running windows 7 and a few running windows 8.1.

Appreciate any further help if you could Matt,

thanks

Hanif

[Matt_Fraser]

Ok. Well, the location of the logs for the installer will be something like C:\Program Files\sapinst_instdir\NW740SR2\MSS\INSTALL\STD. The log analyzer in SWPM is probably already looking here, but if it's easier for you to use Windows Explorer, you can just navigate here directly. That can be helpful for things like the most recent "Date Modified" timestamp, etc.

Logs here that will likely be of most interest will be dev_bootstrap, perhaps others.

The other place to look for logs is in the instance folders themselves. Have a look in \usr\sap\<SID>\J00\work. Dev_server0 should have the startup trace for the server0 instance, which is what seems to be failing. That should provide some more clues.

[Former Member]

Hi Hanif,

>>this installation has worked for my fellow colleagues some running windows 7 and a few running windows 8.1.

Interesting!

I never knew that we can perform an installation(NW7.4) on a client Operating System(Win-8).

Okay, now coming to issue - I suspect the issue could be with Virtual Memory.

Did you set the swap memory according to the requirement? If not, please set the swap memory and restart the machine. then try continuing with the old installation.

If the issue still persists, please attach the logs.

thanks,

pavan

[Former Member]

Hi /

As far as the swap space, despite my limited drive size, I've made the virtual memory a whopping 20GB pretty much verbatim as to what the prerequisites checker dictated.    

From what the C:\usr\sap\A01\J00\work\dev_server0 log shows me, the only thing of real significance is:

F  ********************************************************************************

F  *** ERROR => Java node 'server0' terminated with exit code -1501.

F  ***

F  *** Please see section 'Java program exit codes'

F  *** in SAP Note 1316652 for additional information and trouble shooting advice.

F  ********************************************************************************

I looked up the exit codes from (JSTSG)(Kernel)ExitCodes - Java Server Troubleshooting Guide - SCN Wiki and the error 1501 is 'Some of the managers fails to start'

When the installer starts up, the services instance suffixed by '1' shows green, but the main instance suffixed by '0' stays yellow.

Here are my logs:

dev_bootstrap:

https://db.tt/aE5p8DJM

dev_jstart:

https://db.tt/B3hE0BNT

dev_server0:

https://db.tt/lW5H7M7Z

jstart.jvm:

https://db.tt/hU2WwGoV

sapinst_dev.log

https://db.tt/feXZpYPN

SUPPORT.SAR

https://db.tt/Q7kcZcQJ

Thanks so much once again,

Hanif

[Matt_Fraser]

I think the problem might be a TCP port conflict. The SUPPORT.SAR has all the log files, which is quite useful (and overwhelming!). In the dev_server0 we see just that it failed, as you reported. std_server0.out gives a little more detail, in that it reveals the manager that failed, which was ClusterManager. Still don't know why it failed, though.

Then, in the server0 defaultTrace_00.0.trc we see the error "Cannot open Join Port: [50020]" repeated a number of times, leading (I think) to java exceptions about being unable to establish loopback connections. Further down, a causing error seems to be "Address family not supported by protocol family: bind." I believe that is an artifact of the port conflict.

So, see if you can find out what else is trying to use port 50020.

[Former Member]

Hello Hanif,

This could be an issue with Port No.

Follow the below note for the same.

1628335 - J2EE System fails with "exitcode -1501"

Also, could you check whether this port no is being used anywhere.

The below note will help you in this regard.

1840741 - J2EE System failed with error "Cannot open Join Port"

P.S. Look in services file as well.

regards,

pava

[Former Member]

Matt/Pavan, thanks for your replies....   so indeed my default trace file shows that it cannot open join port 50020 and 'Unable to establish loopback connection'  however there is nothing occupying port 50020.   Issuing the following: I838424@YYZN00548747A:~$ netstat -ano | findstr 50020

returns nothing based on Note 1840741:J2EE System failed with error &quot;Cannot open Join Port&quot;

So nothing seems to be occupying that port!

I also looked at note Note 1789943:Cannot open Join Port. Address already in use

where it suggests using the configtool to change the port manually (as a last resort).  When trying that on this 'partial install' of mine, I get:

Connecting to database ... Default DB connection failed

  FATAL: Main class "com.sap.engine.configtool.console.ConsoleConfigTool" cannot

be started:

  FATAL: java.lang.NullPointerException: while trying to invoke the method com.s

ap.engine.configtool.data.ClusterData.getTemplates() of a null object loaded fro

m static field com.sap.engine.configtool.console.ConsoleConfigTool.clusterData

Any further help is much appreciated.....