Maximum data volume that can flow through PI 7.1

[Former Member]

Hi All,

I have a requirement to push around 10 -12 crores of records from MS Sql server to SAP ISU using PI 7.1in 8 days !!!!!!!!!!!!!I have been told that the hardware sizing done by the BASIS team is the best that Pi 7.1 supports

Please advice me on the following options:

1) Will PI 7.1 be able to handle this?

2) How much hours of scheduling would be required to push to data from SQL - Proxy

3) Any other suggestions???

regards,

Piyush

Edited by: Piyush Mathur SAP on Dec 19, 2009 12:34 PM

Edited by: Piyush Mathur SAP on Dec 19, 2009 12:38 PM

[Former Member · Accepted solution]

Hi Piyush,

The performance would depend on the load of the PI system..You can however monitor the performance by various means.

'Performance Monitoring' in RWB takes care of the Pipeline processing time and is exactly the same as the time shown on SXMB_MONI.

The End-to-End Monitoring gives the AE processing time with the IE processing time but unfortunately it does not give the time for pulling the records from the databse.

Here the AE processing time starts from 'Executed query successfully'. We get the total time for the adapter processing in Communication Channel monitoring.

Hence ideally it is a summation of CC Monitoring (SenderReceiver) Performance Monitoring (either in MONI or RWB).

Hope this helps..

[Former Member · Accepted solution]

Hi All,

Thanks a lot for ur inputs.

I was able to monitor my message flow and found out that

: it takes around 13 sec (Adapter eng + Pipeline time) to pass around 4 MB of data.

: it takes around 18 sec (Adapter eng + Pipeline time) to pass around 5.5 MB of data.

I think this performance is pretty good.

thanks again for ur help.

regards,

Piyush

[Former Member]

Hi,

1) Will PI 7.1 be able to handle this?
2) How much hours of scheduling would be required to push to data from SQL - Proxy
3) Any other suggestions???

As you have mentioned, the hardware is the best that 7.1 supports, so I guess PI will support this situation.

You can schedule the polling as 2000-4000 records every 30 seconds. Since you are pulling almost 100-120 mils in 8 days, I guess this do-able. I am doing 5000 records every minute and I am satisfied with the performance (and I don't have the best of the hardware that PI supports )

Also, the indexing should be done on the table, which will help in increasing the performance.

Regards,

Neetesh

[VijayKonam]

I would recommend that you should keep your basis people in loop so that, they can monitor the DATABASE log file size on the PI system. If it grows larger than expected, the schedulers would not delete them until the time specified.

few thousand records every 30 sec is not a bad idea, provided the system has enough memory and hard disk space for all the logs that PI writes for each message processed.

VJ

[Former Member]

Hi All,

Thanks for your inputs.

I have done a manual testing on my end and my findings are as below:

1) To pull 7K records from SQL database and push these to ISU via PI, it requires around 10 sec.

2) If we pull around 7K records in 1 SQL Query stat and schedule this stat to run every 20 sec then we would need around 100 hrs of scheduled time to fetch 12crore records.

I was trying to do this testing via the "Performance Monitoring" of RWB but the help.sap content is insufficient.

I even tried searching on SDN and found out multiple liinks but not a good one yet

regards,

Piyush

[Former Member]

Hi Piyush,

I just wanted to check if you considered Zip-Unzip option to handle such a heavy load.

YOu can refer the following blogs for details:

https://www.sdn.sap.com/irj/sdn/weblogs?blog=/pub/wlg/ 5899

/people/aayush.dubey2/blog/2007/10/10/zip-transfer-unzip-increase-the-performance-of-your-java-abap-applications

/people/pooja.pandey/blog/2005/10/17/number-formatting-to-handle-large-numbers

/people/alessandro.guarneri/blog/2007/02/21/sap-xi-acting-as-a-huge-file-mover

/people/alessandro.guarneri/blog/2006/03/05/managing-bulky-flat-messages-with-sap-xi-tunneling-once-again--updated

/people/udo.paltzer/blog/2008/11/03/real-life-customer-examples-for-high-volume-scenarios-with-sap-netweaver-process-integration

Rgds,

Puneet

[Former Member]

Piyush,

Can you ensure that before using Performance monitoring, you have done the following configurations:

Go to Integration Engine Configuration:

1. Transaction SXMB_ADMIN

2. Select "Integration Engine Configuration"

3. In the "Specific Configuration Data" window, enter category "PERF" (it will appear in the drop-down list). NOTE: Do not click the button labeled "Specific Configuration" (yet).

4. With "PERF" appearing in the text box, click the blue "i" (Info) button to the right of the text box.

5. Scroll all the way down. The job instructions are there. The programs are RSXMS_COPY_PERF_DATA and RSXMS_PERF_REORG, and SAP provides guidance on how to schedule them.

The job names SAP_XMB_PERF_COPY and SAP_XMB_PERF_REORG are SAP's recommended names for those jobs -- you actually have to type those names in; they don't exist.

Note that the same help page tells you what PERF config settings you have to configure. You set those by clicking the "Specific Configuration" button on that screen, and entering the configuration on the next screen.

Hpe it helps.

Rgds,

Puneet

[Former Member]

Any thoughts?????

[Former Member]

Hi

I will be certainly happy to help you but....the Volume Processing is purely depend on few aspects of PI 7.1

- Scenarios Configured to use high Volume Processing features of PI 7.1 like

[Former Member]

Hi

I will be certainly happy to help you but....the Volume Processing is purely depend on few aspects of PI 7.1

- Scenarios Configured to use high Volume Processing features of PI 7.1 like

a. Message Packaging,

b. Queue Prioritization

c. AAE enabled

d. Are there any BPE Configured

- How about sizing of your PI environment?

a. No CPUs, b. Memory, c. No of Java Server nodes, d. No of Application Servers (aka Dialog Instances)

so like wise there many factors to cosider to conclude on your environment capacity in terms of no of messages processed perhour/day etc.

We have capacity to process 1.5 billion messages per hour But we concluded this fighure based on our load and system processing speed.

So I suggest you to go thorugh the below link to understand the realtime user cases scenarios and asses your system capacity

Becuase NO body can assess better than you about your system.

http://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/00982670-748c-2a10-3985-cdd42fb43...

Regards

Sekhar

[Former Member]

Piyush,

This Thread talks about the same :

Gerard

[Former Member]

Piyush,

I have done a scenario in which have learnt that PI can handle ata a max 3 lakhs files at a time.

In your case it is 3crores, so why dont u divide the data in several chunks and then send.

I know in File adapter we can divide the large data into chunks and send. Please let me know if u can divide the data in ur case.

Thanks

Veeru