BODS 4.1 SP1 job server memory leaks

[djuricz]

Hi all,

We are experiencing memory issues on BODS 4.1 SP1 job server (process JobServer.exe).

Once the job server runs out of memory, Data Services Management Console becomes unresponsive and the job serve is unreachable by external agents used for scheduling of ETL jobs.

In the most recent incident, JobServer.exe had  ~12,000 threads and used 99% of physical memory on the machine.

To fix the issue, we had to restart the Server Intelligence Agent service using Central Configuration Manager.

This releases for memory but it quickly fill up again in matter of days.

The job server is hosted on VM running Windows Server 2008 R2 Standard (64-bit) with 4 cores and 20 GB of physical RAM.

Likewise, we have experienced several crashed with core dumps and connection restarts to our Staging database.

We are suspecting that these crashes have been related to above mentioned memory leaks as well.

If anyone has a solution or recommendation on this issue, please advise how we can go about this issue.

I don't know if this is a known issue that has been addressed with a patch update or in BODS 4.2.

Thanks,

Zlat

[former_member653971]

Can you please elaborate how do you calculate the memory consumption of rfc and adpators

[DayaJha]

Hi Ziatibor,

This is common issue in BODS reason for that issue is Server Sizing, Most of the time we are excluding below details regarding BODS activity, So please share the below information that help me to analyse the issue.

• Count of Real Time Jobs
• Memory Consumption of all Real Time Jobs
• Count of Batch Jobs
• Memory Consumption of all Batch Jobs
• Count of Job Server
• Memory Consumption of all Job Server
• Count of Access Server
• Memory Consumption of all Access Server
• Memory Consumption of Database (Oracle,SQL Server, DB2 etc)
  

If you are using SAP as a Source or Target then Please share below information.

• Count of RFC Configured for BODS
• Memory Consumption of all RFC

If you are using Adapter as a Source or Target then Please share below information.

• Count of Adapter Configured for BODS
• Memory Consumption of all Adapter.

If you are using BOE/BOIPS/Tomcat (Or any Web Application) as a Application then Please share below information.

• Count of BOE/BOIPS Services
• Memory Consumption of all BOE/BOIPS Services.
• Memory Consumption of Tomcat Services. (Or any Web Application)

Above information will help you regarding how much memory utilization in your current environment.

Thanks,

Daya

[djuricz]

We are only running batch jobs extracting data from databases, transforming them, and loading them into target databases. No RFC or any other direct SAP connection.

After investigating the issue over last 2 days, it seems that the problem might lay in how BO IPS and DS resources were configured after the installation.

Since Adaptive Processing Server (APS) runs  a single PID (Process ID) and is a container for 20+ different processes (Sub-Services) within the BusinessObjects 4.0 platform.

The APS is a pure Java based process and it is initialized with 1 GB of Java Heap (-Xmx1g). For most environments this is not a sufficient amount of RAM.

It was not configured to handle higher memory demands by individual processes as manual splitting of services in APS was required. 

See this article for more info:

Sizing and Splitting the Adaptive Processing Servers in BOBJ 4.0 | All Things BOBJ BI Blog

New version of IPS 4.1 SP1 comes with System Configuration Wizard which automates this processin CMC.

Unfortunately, it was not available at the time of installation in IPS 4.0 SP4 which is what we use.

It seems that we need to upgrade to new version of IPS and supported DS to use this wizard:

http://events.asug.com/2013BOUC/Influence/8_26_13/0911%20User%20Influence%20Becomes%20an%20SAP%20App...

The IPS 4.1 admin manual suggests the following:

After you install Information platform services, you will likely want to perform essential post-installation configuration, such as choosing a deployment template,and selecting the SAP Business Objects products your organization will use.

To perform this configuration, and to get Information platform services running

in the shortest time possible, run the "System Configuration Wizard".

Important benefits of using the wizard:

• The wizard explains and guides you through the configuration steps that you'll need to do.

• Using the wizard reduces the likelihood of system misconfiguration.

• The wizard configures settings for you, which speeds up system configuration.

You can also start the wizard from the "Manage" area in the CMC. You can rerun the wizard

at any time to adjust your configuration, and you can always use the "Servers" management page inthe CMC to fine-tune any settings, including the settings you made using the wizard.

Templates for different configurations are listed here:

https://help.sap.com/businessobject/product_guides/sbo41/en/sbo41_bip_deploytemplates_en.pdf

I hope this helps people who also encounter this problem.

[DayaJha]

Hi Djuric,

You can split the EIM Adaptive Processing Services according to your BODS needs using System Configuration Wizard, But you also have to calculate memory consumption of your Batch Job, Other wise it will impact your Job Performance..

Thanks,

Daya

[rribeiro]

Djuric,

In CMC you configured "System Configuration Wizard" in installation?

How many Adaptive Job Server are configured?

How many Job Server's are configured?

You can post configuration of parameters in Adaptive Job Server and Job Server?

[djuricz]

I think there is only job server configured using 4 cores on hosting VM.

Where can I find information on parameters in Adaptive Job Server and Job Server?

[rribeiro]

Djuric,

http://yourserver:8080/BOE/CMC>Servers>Server List.

[djuricz]

Here is the server list, they all run on the same VM. As you can see, we are not using adaptive job server and CMC is showing Danger status (Red box), what does that mean?

Do you have any recommendations on what to look at?

[rribeiro]

Djuric,

JobServer.exe use Adaptive Job Server, clone this server for add more server's and check Healt of Central Management Server in logs.

[djuricz]

Can you please explain to me the difference between job server and adaptive job server? How does that precisely have effect on memory management?

If you can refer met on an article about, I would like to get more knowledgeable about it.

Zlat

[rribeiro]

Follow;

Architecture Data Services;

http://sapsimplified.com/sap-bods/architecture-of-sap-businessobjects-data-services-bods/

Architecture BI Platform

https://help.sap.com/businessobject/product_guides/sbo41/en/sbo41_bip_architecture_en.pdf

Follow Administration Guide;

http://help.sap.com/businessobject/product_guides/sbods42/en/ds_42_admin_en.pdf