Unable to extract data from Orgmodeler

[Former Member]

Dear Experts,

I am in middle of OrgModeler configuration and trying to extract data but unable to extract the same and but idoc listener is started and running in Admin Console.  Should i also start the idoc startidocreceiver.sh in OS level.?  Where should i place SAPjco.jar files in OS level.? unable to find exact location to place the file in the nakisa standard document.

Please note, Initially i have setup same ECC DB in OrgModeler, then later i have created stage database and enter StageDB info in adminconsole, is this the cause of below error.

Thanks

Sharif.

[Former Member · Accepted solution]

Some more error log screen.

[StephenMillard · Accepted solution]

Hi Sharif.

Presumably you are deploying OrgModeler 4.3/4.3 SP1 - please not you should always include your system details in your post (see ).

It looks and sounds to me like you haven't finished doing the initial configuration.

For SAP JCo installation see SAP JCo Installation - Components of SAP Communication Technology - SAP Library.  SAP JCo is standard software an pre-requisite.  It has no config to change so you only need to follow the standard instructions rather than any particular configuration specific to a Nakisa application.  Without this component in place I wouldn't expect you to have much luck communicating with your database.

For configuring and running the initial extract for OrgModeler, please work through section 3 in the VSN43_Deployment guide PDF that came with your Nakisa installation files.  It explains all about what listener process to configure, how to start them, etc.

Hope that helps.

Regards,

Stephen.

[Former Member · Accepted solution]

Dear Stephen,

Thanks for your reply. sorry for inconvenience.

Nakisa OrgModeller 4.3SP1.

Yes i have extracted SAPJCO 3.0 into the below location.

/usr/sap/<SID>/J02/j2ee/cluster/apps/Nakisa/OrgModeler/servlet_jsp/OrgModeler/root/WEB-INF/lib

But still should i deploy sapidocjco.jar, this looks like older version.

Yes i am referring to the nakisa standard OrgModeler deloyment document only. i have already completed steps involved in that till 3.7.  but here my doubt is from step 3.8.

should i edit both startidocreceiver.sh and idocreceivercmd.sh, i have edited startidocreceiver.sh but still command line extractor not starting.

Also bit confused with sapextractor.sh. mentioned in admin guide.

Thanks

Sharif

[StephenMillard · Accepted solution]

Hi Sharif.

sapidocjco.jar and sapjco.jar are separate pieces of software so the version number mis-match has no relevance here.

The idocreceivercmd.sh script is called with 6 parameters to start the receiver (do not edit this file).

The startidocreceiver.sh script calls the idocreceivercmd.sh script and passes in the 6 parameters.  You should edit the startidocreceiver.sh script file.

Modify it to point to the location of your idocreceivercmd.sh, the path to the Java SDK, the path to OrgModeler, the path to SAPJCo, the path to sapidocjco.jar, the path to sapidoc.jar and the path to the required NakisaFramework file.  See the idocreceivercmd.sh script comments for details on what each is and example paths.

With regards to it not starting, you usually get output on the command line when running the script.

Can you please...

1. Provide the contents of the startidocreceiver.sh script.
2. Provide the output from the script when you run it on the command line.
3. Confirm that each of your file path parameters is correct - i.e. the requisite file/s is/are there.
   

Regards,

Stephen.

[Former Member · Accepted solution]

Dear Stephen,

Thanks for your update.

startidocreceiver.sh

Error while starting startidocreceiver.sh

[Former Member · Accepted solution]

Hi stephen,

when i check the log, i noticed the below log is generating while executing startidocreceiver.sh

[StephenMillard · Accepted solution]

Hi Sharif.

The "^M:  not found" error suggests that you have edited the startidocreceiver.sh script in Windows rather than *nix.  Windows uses CR (carriage return) + LF (line feed) as a new line indicator whereas UNIX based systems use just an LF character.  When *nix systems try to process the CR they tend to fail with exactly this sort of error.

If your system has the dos2unix tool installed just pass your script file in as a parameter to strip the CR characters.

If you don't have it installed and want to edit via Windows, many advanced text editors (not Notepad) have the option built into convert line endings, or you could use a script to switch the line endings (I created one for use at a site where the client had some very strict access restrictions).

Regards,

Stephen.

[Former Member · Accepted solution]

Hi stephen,

Yes correct, i have converted the file using dostounix tool.

Now i am able to run the command ./startidocreceiver.sh without ^M but the below output looks like some error again.

[StephenMillard · Accepted solution]

The first line after the error says NoClassDefFoundError ... so it is having trouble finding one of the Java classes.  You didn't confirm the parameter file paths earlier so that is something you absolutely need to do here.

According to the script and the parameters you are passing in you should have the following files at these locations - [[please check this]]:

/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/java
/usr/sap/EP1/J02/j2ee/cluster/apps/Nakisa/OrgModeler/servlet_jsp/OrgModeler/WEB-INF/lib/sapjco.jar
/usr/sap/EP1/J02/j2ee/cluster/apps/Nakisa/OrgModeler/servlet_jsp/OrgModeler/WEB-INF/lib/sapidocjco.jar
/usr/sap/EP1/J02/j2ee/cluster/apps/Nakisa/OrgModeler/servlet_jsp/OrgModeler/WEB-INF/lib/sapidoc.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/IDocReceiverCmd.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/db2jcc.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/db2jcc_license_cu.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/commons-beanutils.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/commons-codec-1.3.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/commons-collections.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/commons-dbcp.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/commons-digester-1.8.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/commons-fileupload.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/commons-httpclient-3.1.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/commons-io.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/commons-jelly-1.0.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/commons-jexl-1.0.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/commons-jxpath-1.2.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/commons-lang.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/commons-logging.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/commons-pool.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/core-renderer.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/dom4j-1.5.2.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/el-impl-1.0.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/FrameworkBin.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/jsf-api.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/jsf-facelets.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/jsf-impl.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/ldapbp.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/log4j-1.2.15.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/ojdbc5_g.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/pd4ml.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/sapdbc.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/serializer.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/spring-ldap-1.2.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/spring.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/sqljdbc.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/ss_css2.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/trinidad-api-1.2.11.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/trinidad-impl-1.2.11.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/webrendererse.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/xalan.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/xercesImpl.jar
/sapmnt/EP1/exe/jvm/hpia64/sapjvm_6.1.076/sapjvm_6/bin/WEB-INF/lib/xml-apis.jar

[Former Member · Accepted solution]

hi stephen,

I am able to start the listener in Nakisa AdminConsole from the begining after i configure orgmodeler.

But unable to start via commandline extractor as i mentioned in the above post.

[StephenMillard · Accepted solution]

... but you still need to check the file paths.  AdminConsole isn't using those file paths.  The command line script is. 

The error message says it isn't finding the Java content it is looking for.  The file paths include the JAR files that contain the java content so currently the error message is suggesting to me that one or more of those files might not be in the right place.

Please check the paths I provided to verify that they are all in place or which are missing/located elsewhere.

[Former Member · Accepted solution]

hi Stephen,

i have all the jar files that you mentioned is located in below

/usr/sap/EP1/J02/j2ee/cluster/apps/Nakisa/OrgModeler/servlet_jsp/OrgModeler/root/WEB-INF/lib

sapjcojar files

/usr/sap/EP1/J02/j2ee/cluster/apps/Nakisa/OrgModeler/servlet_jsp/OrgModeler/root/WEB-INF/lib

[StephenMillard · Accepted solution]

Okay, take a look at IDoc LIstener failed for OrgModeler | SCN

In this particular case the initial JCoIDocHandlerFactory seemed to be related to the SAP JCo installation.  So it might be worth double checking the install to ensure you've added the library path and class path environment variables and that you can run the "java –jar sapjco.jar" test to review the installation.

[Former Member · Accepted solution]

Hi Stephen,

As per SAP note 636912, I have verified and tried reinstalling the sapjco3.jar and sapidocjco.jar and set the environment variable as per installation.html in http://service.sap.com/connectors

env:

SHLIB_PATH = <jco installation path>

CLASSPATH = <jco installation path>/sapjco3.jar.

Also executed, echo $SHLIB_PATH.  variables pointing to target sapjco installation folder.

But stuck up here and unable to execute ./startidocreceiver.sh

[StephenMillard · Accepted solution]

Did you successfully run the sapjco command I posted previously and get the window verifying the successful SAP JCo installation?

[Former Member · Accepted solution]

hi stephen,

Issue has been fixed.

Actual issue at idocreceivercmd.sh file

I noticed, by default sapjco.jar mentioned in idocreceivercmd.sh file is wrong version for installed sapjco 3.0. so i just edited idocreceivercmd.sh and update the correct version as below.

changed sapjco.jar to sapjco3.jar and sapidoc.jar to sapidoc3.jar. now i am able to run

commandlineextractor > ./startidocreceiver.sh in OS level successfully.

Thanks for your support and input.

One more doubt here.  but still i can see some warning message as below while starting

./startidocreceiver,sh.

baterp1:ep1adm 70> ./startidocreceiver.sh &&

log4j:WARN No appenders could be found for logger (org.springframework.jdbc.datasource.DataSourceUtils).

log4j:WARN Please initialize the log4j system properly.

But in the AdminConsole i can see the below message.

08 Sep 2015 20:46:48 INFO com.nakisa.IDocLogger - IDocServer is listening now...

Thanks

[Former Member · Accepted solution]

Hi Stephen,

As per the deployment guide i have completed the idoc initial load via command line extractor and PFAL in SAP GUI.

Then i have executed Table extraction in Admin Console, after the executiion i can see the message in the same screen its completed.

But in the debug tool > log manager still it shows in creating and inserting HRP1001 HRP1000 etc...

Thanks

[StephenMillard · Accepted solution]

That's odd.  It should have shown up when I asked you to check "/usr/sap/EP1/J02/j2ee/cluster/apps/Nakisa/OrgModeler/servlet_jsp/OrgModeler/WEB-INF/lib/sapjco.jar" earlier.  Still at least you've got to the bottom of it now

The cds.log generation uses the log4j component (an Apache logging module).  For the command line logging sometimes the initial file copy that takes place doesn't copy the log properties files in (I've never spotted a particular pattern of when it works and when it doesn't).  The first run however almost always misses the properties.

Check in the .../root/WEB-INF/lib/log directory for the *log*.properties files that exist in the .../root/log directory (forgive me if my upper/lower case path references donot match exactly).  If they do, try restarting the listener and see if you still get the error.  If they are not there, copy them in and restart the listener.

[Former Member · Accepted solution]

hi stephen,

Attached cds.log and the below log screen.  as per the below screen table not exist Where?

Is it in Destination DB (Staged DB)?

One thing i wanted to update here.  initially when i installed orgmodeller as i mentioned in my previous scn query. ()  i have configured wrongly to ECC DB and then removed immediately , then later i have created staged DB and then pointed that.

thanks

cds.log.zip

[StephenMillard · Accepted solution]

You need to work back through the log and identify the first error.  I can see the one immediately prior had a table/view does not exist just like that one.

[StephenMillard · Accepted solution]

I can see the following in the log:

07 Sep 2015 21:52:42 ERROR com.nakisa.Logger  - com.nakisa.framework.sql.ConnectionManager : Error creating SQL data connection

07 Sep 2015 21:52:42 ERROR com.nakisa.Logger  - java.sql.SQLException: ORA-01005: null password given; logon denied

Are you sure your database connection is configured correctly?  It looks like it is saying you haven't entered a password into your data connection.

[Former Member · Accepted solution]

Hi stephen,

Yes I have configured database connection as per guide.

Below are the location I have entered staged DB connection details.

1. 1. SAP ALE/IDOCS/Extractor Settings > Configure SAP ALE/IDOCS/Extractor > Destination.  Test connection successful.
2. 2. Application-wide Settings > Scenario Connection Settings> test connection is successful.
3. 3. Org Chart > Hierarchies > Org Unit > Data Connection > test connection successful.

thanks

[Former Member · Accepted solution]

Hi Stephen,

I have attached the latest log which generated immediately after i execute extraction in Admin console  SAP ALE/IDOCS/Extractor Settings > Configure SAP ALE/IDOCS/Extractor > Start Extraction.

I would like to know whether table HRP1000 / HRP1001 will automatically creates in OrgModeler Stage DB or how it's work.?

If you check the attached error log, it is saying table doesn't not exits for eg: the below.

09 Sep 2015 15:13:35 ERROR com.nakisa.Logger - : org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar [DROP VIEW V_HRP1001_JobToJobFamily]; nested exception is java.sql.SQLSyntaxErrorException: ORA-00942: table or view does not exist

2.

3. 09 Sep 2015 15:13:35 ERROR com.nakisa.Logger - org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar [DROP VIEW V_HRP1001_JobToJobFamily]; nested exception is java.sql.SQLSyntaxErrorException: ORA-00942: table or view does not exist

4.

5. Caused by: java.sql.SQLSyntaxErrorException: ORA-00942: table or view does not exist

But i have also checked in staged DB some tables like start with T* / Heartbeat / scenario etc.. all created already. but i am not able to get table HRP1000 / HRP1001 in staged DB. Correct me if i am wrong here

thanks

[StephenMillard · Accepted solution]

I think these tables should have been created in your initial load, so perhaps that didn't run as well as expected?

Is there anything notable appearing in either of these log files?

• .../root/idoc/log/idoc.log
• .../root/WEB-INF/lib/log/cds.log

Please review any older logs too should a log roll have occurred.

You mention that you carried out the initial data load and PFAL.  Did you follow this with the check table extraction and join execution?

Regards,

Stephen.

[Former Member · Accepted solution]

HI stephen,

I have followed check table extraction and join execution after initial load steps.

I have a doubt regarding initial load, when i executed "./startidocreceiver.sh" in OS level (HP-UX).  it was keep running... so i killed that and executed with background option like below

"./startidocreceiver,sh &&" to run in background,  Is this correct?, or should run without &&.

i will check the logs in the below and come back with my findings.

• .../root/idoc/log/idoc.log
• .../root/WEB-INF/lib/log/cds.log

thanks

[StephenMillard · Accepted solution]

It is a non-interactive script so you can run it how you like.  I'd be tempted to pipe the output to a file if it wasn't run as a foreground process.  That way I could still get to any standard output data.

I'm not sure exactly what shell you're using, but normally the script would end ".sh" rather than ",sh" and one "&" would be used to run it as a background process ("&&" being used for command concatenation).

What you wouldn't want to do is kill the process.  The trigger script process I believe is the one that is also running the process for the actual receiver.  Closing it kills both.  I guess you could modify the scripts to change the behaviour, but remember sometimes there's useful stuff piped to the standard output and it is always worth trying to make it available for debugging purposes.

[Former Member · Accepted solution]

Hi stephen,

Sorry it was typo mistake.  It is actually "./startidocserver.sh &"

I didn't understand the below,

" I guess you could modify the scripts to change the behaviour, but remember sometimes there's useful stuff piped to the standard output and it is always worth trying to make it available for debugging purposes ".

thanks,

[Former Member · Accepted solution]

Hi Stephen,

Issues has been fixed.

For all the errors that in cds.log i was facing, it is due to the improper sequence of the events and setup.

Solution:

1. Reset the current build

2. Followed the steps with same sequence in the OrgModeler Deployment / Admin Guide.

Thanks

Instruction to follow Orgmodeler Initial setup.txt.zip