New Portal App Server / Dialog Instance is missing content

[jill_diesman]

Dear all,

We have a problem with a newly installed Portal application server / Dialog Instance: it is missing all applications in
/usr/sap/EPQ/J00/j2ee/cluster/server0/apps.

Background:  Our QA Portal environment has had only one instance for many years; it is currently on NetWeaver 7.31 SP09.  I recently installed a new Portal application server / Dialog Instance (call it "Server2"), using the installation guide "SAP Systems Based on the Application Server Java of SAP NetWeaver on Windows: MS SQL Server Using Software Provisioning Manager 1.0."  The installation went smoothly.

I can start and stop Server2 normally, and can log on to it as well, but applications like ESS, Business Intelligence, etc. are failing.  At the file system level Server2 is missing numerous files and directories. Following are some examples:

• /usr/sap/EPQ/J00/j2ee/cluster/server0/apps
  Expecting 3 directories:  com.adobe, local, sap.com.
  What's in Server2:  only the sap.com directory
• /usr/sap/EPQ/J00/j2ee/cluster/server0/apps/sap.com
  Expecting about 1,225 directories.
  What's in Server2:  only the irj directory
• /usr/sap/EPQ/J00/j2ee/cluster/server0/apps/sap.com/irj
  Expecting 2 directories:  PortalRuntimeContainer, servlet_jsp.
  What's in Server2:  only the servlet_jsp directory
• /usr/sap/EPQ/J00/j2ee/cluster/server0/apps/sap.com/irj/servlet_jsp
  Expecting 2 directories and 2 files:  root and work directories, epbc.war and web_crc.properties files.
  What's in Server2:  only the root directory
• /usr/sap/EPQ/J00/j2ee/cluster/server0/apps/sap.com/irj/servlet_jsp
  Expecting 3 directories and about 5 files:  META-INF, portalapps, and WEB-INF directories.
  What's on Server2:  only the WEB-INF directory

Any help or advice about these missing files and directories would be very much appreciated.

Regards,

Jill

[Former Member · Accepted solution]

Force redeployment from the DB.

regards

[jill_diesman · Accepted solution]

Hi Lawrence,

Thank you for your suggestion.  I am not familiar with forcing redeployment from the DB.  Can you let me know how to do this or point me to the online help?

Thanks for your help,

Jill

[Former Member · Accepted solution]

I didnt tryed on my own, since its very deep BASIS (and you shouldnt also, unless you have appropriate expertise, so better ask your BASIS consultant), but it must be something like:

• stopsap J2EE
• sapcontrol -nr 01 -function StopService
• sapcontrol -nr 02 -function StopService
• cleanipc 01 remove
• cleanipc 02 remove
• /usr/sap/XXX/JXX/j2ee/cluster/bootstrap/bootstrap.properties
  set element.resync=FORCE
• sowie im /usr/sap/XXX/JXX/j2ee/cluster
  rename directories apps, server0, server1 to *.old
• startsap

cheers

[hemanth2 · Accepted solution]

Hi Jill,

Such an issue can be caused due to inconsistencies and it will be worthwhile to run the force bootstrap once (perhaps when you are scheduling the next offline backup on the server).

1)

Stop the j2ee server. Under cluster directory (/usr/sap/<SID>/J<nr>/cluster) rename server0 and subsequent nodes to ".old" respectively. The intention here is to create a backup of the existing files. Do not rename the bootstrap folder as this is needed for the sync process.
2)

Change bootstrap.properties file to ensure it has element.resynch=force as per SAP note 710663.

3)

Restart the J2EE Engine. The new server folders should be created during the bootstrapping and this is a time consuming process as all the contents of the server nodes will be syched with the DB content.

4)

Remove the element.resynch=force property or change it to element.resynch=detect which is the default setting.

Cause

In an SAP J2EE Engine, all the data as well as files/binaries are stored in the database (DB) level along with the file system level. So, whenever there is any change in any of the binaries/files, the DB recognizes the same and synchronizes it with the file system. The synchronization happens in the bootstrap phase of the engine startup It is possible that the files are corrupt on the file system level which leads to the server node being in an inconsistent state. Issues can also happen because the files are never synchronized from the DB to the file system.
 

Thank you!

_____________

Kind Regards,

Hemanth

SAP AGS
 

[jill_diesman · Accepted solution]

Lawrence,

Thank you for your reply.  I have been on vacation and unable to work on this, however I will be following your and Hemanth's advice as soon as I can schedule some downtime for the system.

Thank you very much!

Regards,

Jill

[jill_diesman · Accepted solution]

Dear Hemanth and Lawrence,

I followed your directions but am back at the place from which I started.  Here are the steps I followed:

1) Stopped the J2EE server only on the Application Server with the problem.  Renamed server0 to server0.old (this is the only server node).  Did not rename the apps folder as Lawrence had suggested since there doesn't appear to be a problem with this folder's contents.

2) Modified bootstrap.properties file as recommended (changed element.resynch=FORCE).

3) Restarted the J2EE engine.  A new server0 folder was created, but it was not completely synched with the database content.  The content that was created was the same content I referenced in my original post.

4) Modified bootstrap.properties to revert change made in step #2  (changed back to element.resynch=DETECT).

Do you have any other ideas or suggestions?

Many thanks,

Jill

[hemanth2 · Accepted solution]

Hi Jill,

It is then clear that the data is not available in the DB as well. In this case you will need to redeploy the SCAs again (identify which component is missing, download the latest patch from SMP) and then deploy it using one of the tools mentioned in sap note 1715441.

Thank you!

_____________

Kind Regards,

Hemanth

SAP AGS

[jill_diesman · Accepted solution]

Dear Hemanth,

Our first Portal Application Server in this environment, and its database, have been up and running without incident for 4-5 years, so I think the data is available in the database.  Therefore I will try one other test before following your latest recommendation.

I will post the results of my test shortly.

Best Regards,

Jill

[hemanth2 · Accepted solution]

Great news Jill, please keep us posted

[jill_diesman · Accepted solution]

I decided to copy the missing files from our original/first application server to our new application server.  The directories and files I copied were in /usr/sap/EPQ/J00/j2ee/cluster/server0/apps.  I knew this would not negatively affect the original/first application server, so I thought it was worth a try.

I also found out I needed to have some new firewall rules written for the new application server.  So far things have been working well.

Thank you Hemanth and Lawrence for your help, especially in regard to forcing a redeployment from the database.

Best regards,
Jill

[former_member278126]

we had same issue with one of the application server was missing directories. we just moved the apps directory to apps.old and restarted the instance . It forced the instances to build the apps direcotory completely . After a complete restart the apps direcotry was fine .

[Former Member]

We are also facing similar issue like Jill. This is happening during the installation app server NW 731, MS SQL Server. Installation is failing at the step 'Create Mesage Server Access Control List'. After looking at dev_jstart, it was found that the directory 'apps' is missing. I have stopped both CI, App server. Copied the missing 'apps' directory from CI to App server and able to install the app server successfully. Will this kind of manual copy create any problems in future? I have n't seen this error during earlier installations of app serer.

Thanks,

Krishna Vaka

[jill_diesman]

Hi Krishna,

I have not run into any problems since copying those directories and files from the primary application server (CI) to the secondary application server.  this year I even performed a system copy to install new hardware for all the instances in that environment, and everything went fine.

Regards,

Jill

[Former Member]

Hi Jill,

Thanks for the information. In my case several re-installs have not helped me. Faced the same issue in DEV as well as QA app server installations. Finally found your thread and copied the apps directory (and its sub directories) from CI-DB to application server. Apart from this I have also, manually maintained the host name in hosts files. is this a bug in NW 731 (Java) MS SQL server installation?

Regards,

Krishna Vaka

[jill_diesman]

Hi Krishna,

I don't know whether or not this is a bug.  And unfortunately I don't have any further experience with the same scenario because we have since upgraded all our Portal/Java systems to NetWeaver 7.4.

Regards,

Jill

[Former Member]

Hi Jill,

Try the below link, maybe it helps.

http://scn.sap.com/community/java/blog/2013/09/26/deploy-sap-java-applications

Regards,

Sushil

[jill_diesman]

Thank you, Sushil. I will read through this.

Regards,

Jill