SIA restart loop and infinite hprof files

[Former Member]

Hello BO Experts,

Im having a problem with SIA, its just not starting(it never appears as stopped but it continues initializing and starting and then intializing), and generating hprof files in the logging folder until the disk is full.

What I have done already:

Changed the java allocated memory on setenv.bat file to:

set JAVA_OPTS=%JAVA_OPTS% -Xmx4096m -XX:MaxPermSize=512m

Read the heap files on the Memory Analyzer, where all the files show the same error:

Problem Suspect 1

The thread java.lang.Thread @ 0x16acbf80  Thread-0 keeps local variables with total size 268 450 440 (68,23%) bytes.

The memory is accumulated in one instance of "char[]" loaded by "<system class loader>".

Repaired the BO Installation with no results

Increased the server memory to 16GB.

BO information:

BO 4.0 SP4 with data integration.

Please if anyone have any hint about what to do I would really appreciate. this error happened out of nothing,

Thanks on advance.

Santiago Silva.

[Former Member · Accepted solution]

Hi,

Please check clusterinfo.1400.properties located in c:\.businessobjects folder. If you are not able to find the file just search the computer with the filename.

The file size should be of few KBs. There was a known issue when this file use to get corrupted and grow upto GBs causing the OutofMemory exception.

If the size is more, just rename the file and try to start the SIA, this should create a new file. Once this works fine, you can get rid of the huge file.

Hope it helps.

Regards

Chinmaya

[Former Member · Accepted solution]

Hello Chinmaya,

Thanks for your answer, I checked that, and file is about 70KB so nothing too BIG.

I renamed the file just in case and started SIA but no new clusterinfo.1400.properties was generated.

I'm a little confused now.

Thanks,

Santiago.

{EDIT}

Acutally I was looking in the wrong path, there are 2 of this files, one under windows/users/theuser and one under c.

Now my services are all UP!, no heap files .

Chinmaya you are my hero!! Thank you a lot!!!!!!

And thanks to Ilias Fytrakis and Ritesh Sharma, you were really helpful too!!!!

[Former Member]

Hello again Santiago,

Some other things that I would check:

1.) Do you have enough free space on the repository database? (please give us some information about the database - RDBMS used etc.)

2.) If you stop the Server Intelligence Agent service, are there any files in <INSTALLATION_PATH>\SAP BusinessObjects Enterprise XI 4.0\serverpids directory?

Thanks,

Ilias

[Former Member]

Illias!

1-Yes I have enough free space.

Its an sql server 2008 R2, on separated server.

2-Yes! this files keep there.

Any suggestions?

Thanks!

[Former Member]

Good morning Santiago,

When you stop the Server Intelligence Agent (SIA) service, the serverpids directory should be empty. If it's not, then remove those files and try to start SIA again.

Regards,

Ilias

[Former Member]

Good morning Ilias, I've deleted the files and started again SIA, same result, with no PIDS generated.

Regards,

Santiago

[Former Member]

Hello,

I suppose that the DSN's created to provide connectivity with MS SQL Server 2008 R2 are working fine (try to test each connection - Repository & Audit database). Also you could use Process Explorer to see more details about SIA.exe

Thanks,

Ilias

[Former Member]

Thaks Ilias, that's a good hint!

What I can see on the process explorer is that dependant services like CMS, JobServer, etc from SIA are NEVER initialized, SIA just keeps starting and stopping.

Here is the printscreen.

[Former Member]

What about the DSN connections (ODBC Data Source Administrator)?

[Former Member]

ODBC are all OK

[Former Member]

Do you have the auto-expand option active for audit & repository databases? I think that by default this setting is activated in MS SQL Server. Additionally, do you get any useful information from the error logs located in the logging directory of BI installation?

I'm all out of ideas right now.

Have a nice weekend,

Ilias

[Former Member]

Here is my SIA_JVM.log file, maybe u can help me to understand it:

Option is not recommended for use in productive environments: -XX:+ExitVMOnOutOfMemoryError

JVMX version - May  3 2012 13:12:37 - 61_REL - optU - windows amd64 - 6 - bas2:171636 (mixed mode)

(CompilerOracle read from file C:\Program Files (x86)\SAP BusinessObjects\SAP BusinessObjects Enterprise XI 4.0\win64_x64\sapjvm\jre\.hotspot_compiler)

java.lang.OutOfMemoryError: Java heap space

Dumping heap to C:/Program Files (x86)/SAP BusinessObjects/SAP BusinessObjects Enterprise XI 4.0/logging/\java_pid3220.hprof ...

Heap dump file created [396823233 bytes in 10.575 secs]

   :    [1936]  15:08:46    ***ERROR (vmError.cpp:1656): java.lang.OutOfMemoryError: Java heap space

"Thread-0" cpu=19203.72 [reset 19203.72] ms allocated=2166905032 B (2.02 GB) [reset 2166905032 B (2.02 GB)] defined_classes=932

io= file i/o: 508585467/0 B, net i/o: 0/0 B, files opened:36, socks opened:1  [reset file i/o: 508585467/0 B, net i/o: 0/0 B, files opened:36, socks opened:1 ]

prio=6 tid=0x0000000057c24490 nid=0x790 / 1936 runnable  [_thread_in_vm (_running), stack(0x0000000058690000,0x00000000586d0000)] [0x00000000586ce000]

   java.lang.Thread.State: RUNNABLE

  at java.util.Properties$LineReader.readLine()I(Properties.java:466)

  at java.util.Properties.load0(Ljava/util/Properties$LineReader;)V(Properties.java:337)

  at java.util.Properties.load(Ljava/io/InputStream;)V(Properties.java:325)

  - locked <0x0000000017016c58> (a com.crystaldecisions.celib.io.PersistentConfiguration)

  at com.crystaldecisions.celib.io.PersistentConfiguration.refresh()V(PersistentConfiguration.java:57)

  at com.crystaldecisions.enterprise.ocaframework.PersistedClustersInfo.readPreferences()V(PersistedClustersInfo.java:299)

  at com.crystaldecisions.enterprise.ocaframework.PersistedClustersInfo.<clinit>()V(PersistedClustersInfo.java:68)

  at com.crystaldecisions.enterprise.ocaframework.APSServerHandler.<init>()V(APSServerHandler.java:69)

  at com.crystaldecisions.enterprise.ocaframework.ServerController.<init>(Lcom/crystaldecisions/enterprise/ocaframework/ServiceMgr;)V(ServerController.java:125)

  at com.crystaldecisions.enterprise.ocaframework.ServiceMgr.<init>()V(ServiceMgr.java:418)

  at com.crystaldecisions.enterprise.ocaframework.ServiceMgrFactory.getServiceMgr()Lcom/crystaldecisions/enterprise/ocaframework/IServiceMgr;(ServiceMgrFactory.java:66)

  - locked <0x0000000011fb75a0> (a java.lang.Class for com.crystaldecisions.enterprise.ocaframework.ServiceMgrFactory)

  at com.businessobjects.cad.CADServant.init(Lcom/businessobjects/cad/interfaces/IProcessManager;Lcom/businessobjects/cad/interfaces/IInfoStoreAgent;Lcom/businessobjects/smartservers/CommandLineConfig;)I(CADServant.java:54)

  at com.businessobjects.cad.ClusterAgent.init(Lcom/businessobjects/cad/interfaces/ICAD;Lcom/businessobjects/smartservers/CommandLineConfig;)I(ClusterAgent.java:40)

  at com.businessobjects.cad.CAD.init(Lcom/businessobjects/smartservers/CommandLineConfig;)I(CAD.java:40)

  at com.businessobjects.cad.CADmain.main([Ljava/lang/String;)V(CADmain.java:236)

   :    [1936]  15:08:46    ***ERROR (vmError.cpp:1663): Exiting VM because ExitVMOnOutOfMemoryError was set...

[Former Member]

Good morning Santiago,

Please have a look at the following article which describes how to analyze memory dumps

Installing Eclipse Memory Analyser Tool for Heap Dump Analysis | SAP NW Newbie

Regards,

Ilias

[Former Member]

Hello Ilias,

I've already used that tool and nothing that can give me a clue about how to fix this.

Thanks for your answer, If you have any other suggestion please let me know.

Regards,

Santiago.

[former_member190982]

Hi Santiago,

(I am Assuming that your env is on windows.)

To troubleshoot this further, forceterminate the SIA(which is in running or stopping status)

If its does not stops, you may kill the SIA.exe

The right click the SIA and add (space ) -trace at the end of the command line.

Then try to start the SIA.

There will we a tarec file generated in the logging folder.

Attach the logs as well the eventviewer logs.

Regards,

RVS

[Former Member]

Hello Ritesh,

Here I've done what u said and here is the trace file.

Thanks!

SIA_SIA_trace.000002.txt.zip

[former_member190982]

Hey Santiago,

I have gone through the tarce but it does not show any error.

I have few questions:

1. Is the SIA started? can you log in to CMC or BI Launchpad:?

2.Have you splitted the Adaptive processing servers(APS):?

3. Or is it that you SIA is does starts but it goes in to stopping then intializing ->starting status randomly and you see hprof file when this happens.

if your case is ' 3 ' , then we will have to find out which server is causing the hprof files to be generated, due to which you might be seeing the issue(stuck in initializing/starting status)

Try this :

Stop the SIA.

Right click property->in the command line at the end add the below switch.

-noautobootservers

This switch will only start tye SIA and CMS and no other services.

Start the SIA.

Keep this for a while and let us  know your observations.(do you still see the hprof, status of SIA etc)

Regards,

RVS

[Former Member]

Hello Ratesh,

The scenario is number 3.

I tried the noautobootservers but it keeps generating the hprof files and SIA keeps starting/stopping.

Weird isn't it?

Santiago.

[Former Member]

Dear Santiago,

*.hprof files are heap dumps generated by java.exe generally due to OutOfMemoryError exception.

There are several SAP Notes regarding your case such as:

• 1768898 - *.hprof files filling up disk space in XI 4.0
• 1769420 - Adaptive Processing Server fails frequently
• 1900005 - Error "Object failed to run due to an error while processing on the Job Server" when scheduling a large webi document to CSV format in BI 4.0
• 1632699 - Large *.hprof files filling up disk space in XI 4.0
• 1949549 - SIA not starting on Linux Server

In your case I would check the following:

1.) Which user starts the SIA windows service? Is he a local or domain user? Does he belongs to the group of administrators?

2.) Have you assigned this user with 'Act as part of the operating system' security policy?

3.) Do you use XX:+HeapDumpOnOutOfMemoryError in Adaptive Processing Server's configuration?

Also have a look at the logging directory of the SAP BI installation folder for more information.

I hope this helps you a bit.

Regards,

Ilias

[Former Member]

Thanks for the answer Ilias,

I have already read that notes, but no one applies to mi case.

The service is started by the local administrator.

The HeapoDumpOnOutOfMemoryError is activated yes, but I cant get to the CMC so I can't change the configuration, or can I?

The loggin folder is what I check every time I try to start SIA and keeps generating heap dumps.

Any other ideas?

Thank u!

Santiago.