Restarting CMS on AIX

[Former Member]

Hey everyone,

We have a BO 4.1 installation SP 2 on a AIX server. The situation is we had lost connectivity to the CMS database and the application was down for a day, now the database connectivity is restored but the CMS has not come up. I have used  the following command to start it up but I am still not able to log in to the the BI LaunchPad.

Commands i have used to startup :

./svn_startup.sh

./startservers

./ccm.sh -start

./ccm.sh -start -cms machineIP:6400 -username administrator -password password -authentication secEnterprise

./ccm.sh -start -cms machinename:6400 -username administrator -password password -authentication secEnterprise

I have seen the logs and found the below in the "ccm_20140917_1410945693737.log"

Wed Sep 17 2014 12:21:33 GMT+0300 (AST) /bo/sap/sap_bobj/setup/jscripts//ccmunix.js started at Wed Sep 17 2014 12:21:33 GMT+0300 (AST)

Wed Sep 17 2014 12:21:33 GMT+0300 (AST) About to process commandline parameters...

Wed Sep 17 2014 12:21:33 GMT+0300 (AST) Starting all servers...

Wed Sep 17 2014 12:21:33 GMT+0300 (AST) Starting boprod...

Wed Sep 17 2014 12:21:34 GMT+0300 (AST) Finished processing commandline parameters.

Wed Sep 17 2014 12:21:34 GMT+0300 (AST) Warning: No password supplied. Setting password to default.

Wed Sep 17 2014 12:21:34 GMT+0300 (AST) Warning: No user name supplied. Setting user name and password to default.

Wed Sep 17 2014 12:21:34 GMT+0300 (AST) Warning: No authentication type supplied. Setting authentication type to default.

Wed Sep 17 2014 12:21:34 GMT+0300 (AST) Warning: No CMS name supplied. Setting CMS name to machine name.

Please someone help me start the services back up and running.

[Former Member · Accepted solution]

Guys,

Got the issue sorted with help from SAP Support, however still dont know the root cause of the issue. It got resolved by using adding a node but entering the same details of the original node so it was actually like recreating the node. This was done using the servercofig.sh then "add a node" option

[denis_konovalov · Accepted solution]

this means that when DB went down, the SIA properties in CMS DB got corrupted.
By re-creating the node, you fixed the entries in CMS DB and product started.

[Former Member · Accepted solution]

Yes, seems like. reading the logs seems like the CMS port was occupied or CMS was not found on that port. So basically port configuration got corrupted. Now, I do not understand how can I be assured such an issue wouldn't happen again. It is a production server.

[denis_konovalov · Accepted solution]

work with your DBA on ensuring CMS DB availability.

[denis_konovalov]

run ./cmsdbsetup.sh and verify DB connectivity actually works.

then

run ./stopservers

then run ps -ef | grep sap
If you see any BOBJ processes, kill them.
Then run ./startservers

run ps -ef | grep sap  and see if cmsd process is running.
then try to login.

[Former Member]

Thank you Denis,

I have stop the servers then run ps -ef | grep sap and got the below response. I think there are still some services running. I dont quite understand that. But tomcat is up for sure. Do you need me to stop the tomcat too?

ps -ef | grep sap response:

root  7012550

1   0   Sep 01

- 24:24 /bo/sap/sap_bobj/enterprise

_xi40/aix_rs6000_64/sapjvm//bin/java -classpath /bo/sap/sap_bobj/tomcat/bin/boot

strap.jar:/bo/sap/sap_bobj/tomcat/bin/tomcat-juli.jar -Djava.util.logging.config

.file=/bo/sap/sap_bobj/tomcat/conf/logging.properties -Dbobj.enterprise.home=/bo

/sap/sap_bobj/enterprise_xi40 -Djava.awt.headless=true -Djava.net.preferIPv4Stac

k=false -Xmx2048m -XX:MaxPermSize=384m -XX:+HeapDumpOnOutOfMemoryError -Xloggc:/

bo/sap/sap_bobj/tomcat/logs/tomcat.gc.log -XX:+PrintGCDetails -XX:+UseParallelOl

dGC -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.end

orsed.dirs=/bo/sap/sap_bobj/tomcat/endorsed -Dcatalina.base=/bo/sap/sap_bobj/tom

cat -Dcatalina.home=/bo/sap/sap_bobj/tomcat -Djava.io.tmpdir=/bo/sap/sap_bobj/to

mcat/temp org.apache.catalina.startup.Bootstrap start

bousr  9371702

1   0 17:17:37

-  0:00 /bo/sap/sap_bobj/enterprise

_xi40/subversion/bin/svnserve --root=/bo/sap/sap_bobj/enterprise_xi40 --listen-p

ort 3690 --pid-file /bo/sap/sap_bobj/serverpids/LCMSubversion.pid -d

Appreciate your help,

[denis_konovalov]

tomcat can still run as well as subversion, no need to stop them or kill them.

If you are not familiar with OS commands, please consult with your Unix Admin, administering BOE on Unix requires basic knowledge of UNIX, how to start and kill processes, how to check system logs, directory navigation and file management.

now that all BOE servers are stopped - run cmsdbsetup script and see if connection to CMS DB is working and can return correct cluster key.
If there are problems with that - either CMS DB was corrupted or connection doesn't work.

Let us know what message you get.

[Former Member]

Connection to CMS DB is working. It even allows me to see the cluster key which however is encrypted. I eve tried changing the cluster key and it works and updates

[denis_konovalov]

in that case you can run ./startservers, give it about 10 minutes and try to login

[Former Member]

Started the servers and still not successful. Was reviewing the logs and found this in the cmsdbsetup log file:

Thu Sep 18 2014 15:01:27 GMT+0300 (AST) Warning: No password supplied. Setting password to default.

Thu Sep 18 2014 15:01:27 GMT+0300 (AST) Warning: No user name supplied. Setting user name and password to default.

Thu Sep 18 2014 15:01:27 GMT+0300 (AST) Warning: No authentication type supplied. Setting authentication type to default.

Thu Sep 18 2014 15:01:27 GMT+0300 (AST) Warning: No CMS name supplied. Setting CMS name to machine name.

Thu Sep 18 2014 15:01:27 GMT+0300 (AST) Current cluster key: [[luU6jD4ff7EfEf3F1NVAXA]]

[denis_konovalov]

this is normal log entry.

Please look for cms log or any BOE related system messages.

what error do you get when logging into BILaunchpad ?

[Former Member]

Asked for system log from AIX admin,

BI Launchpad & CMC log in error:

Account information not recognized: Could not reach CMS 'boprod:6400'. The CMS on machine 'boprod' was stopped due to a critical error. (FWM 20031)

I believe a restart will solve the issue however i cannot recommend this to the customer every time such issue is faced. I need to find out why am I unable to startup the services using the commands.

[denis_konovalov]

when you run ps -ef | grep sap  , do you see cmsd process running ?

the error means that cms service has crashed. without knowing why it crashed (we need logs) we won't know why.

I'd suggest opening a support Incident with SAP for better troubleshooting.

[Former Member]

Nope,

cmsd process was not found on running ps -ef | grep sap. I will create a support ticket and will update. thank you for all your help.

[former_member191664]

1. Do you see any CMS database related error on the AIX syslog?

2. Are you able to login to CMS system and Auditing databases from AIX term?

Hope this helps,

Jin-Chong

[Former Member]

Hi Jin,

1. I am new to AIX I do not know how do i check the syslog for AIX.

2. I am able to login to oracle using cms schema from the BO AIX server using sqlplus bocms/bocms@tnsname

Appreciate your help,

Mahee

[TammyPowlas]

Hi - I moved this to the BI platform space for faster response