12 Replies Latest reply: Dec 23, 2009 7:40 AM by Piyush Mathur RSS

Maximum data volume that can flow through PI 7.1

Piyush Mathur
Currently Being Moderated

Hi All,

I have a requirement to push around 10 -12 crores of records from MS Sql server to SAP ISU using PI 7.1in  8 days !!!!!!!!!!!!!I have been told that the hardware sizing done by the BASIS team is the best that Pi 7.1 supports

 

Please advice me on the following options:

1) Will PI 7.1 be able to handle this?

2) How much hours of scheduling would be required to push to data from SQL - Proxy

3) Any other suggestions???

 

regards,

Piyush

 

Edited by: Piyush Mathur SAP on Dec 19, 2009 12:34 PM

 

Edited by: Piyush Mathur SAP on Dec 19, 2009 12:38 PM

  • Re: Maximum data volume that can flow through PI 7.1
    veer kumar
    Currently Being Moderated

    Piyush,

      I have done  a scenario in which have learnt that PI can handle ata a max  3 lakhs files at a time.

    In your case it is 3crores, so why dont u divide the data in several chunks and then send.

     

    I know in File adapter we can divide the large data into chunks and send. Please let me know if u can divide the data in ur case.

     

    Thanks

    Veeru

  • Re: Maximum data volume that can flow through PI 7.1
    Gerard Christopher
    Currently Being Moderated

    Piyush,

     

    This Thread talks about the same : Re: Maximum file Size PI 7.1 Can handel

     

    Gerard

  • Re: Maximum data volume that can flow through PI 7.1
    Piyush Mathur
    Currently Being Moderated

    Any thoughts?????

  • Re: Maximum data volume that can flow through PI 7.1
    Currently Being Moderated

    Hi,

     

     

    1) Will PI 7.1 be able to handle this?

    2) How much hours of scheduling would be required to push to data from SQL - Proxy

    3) Any other suggestions???

     

     

    As you have mentioned, the hardware is the best that 7.1 supports, so I guess PI will support this situation.

     

    You can schedule the polling as 2000-4000 records every 30 seconds. Since you are pulling almost 100-120 mils in 8 days, I guess this do-able. I am doing 5000 records every minute and I am satisfied with the performance (and I don't have the best of the hardware that PI supports )

     

    Also, the indexing should be done on the table, which will help in increasing the performance.

     

    Regards,

    Neetesh

    • Re: Maximum data volume that can flow through PI 7.1
      Vijayashankar Konam
      Currently Being Moderated

      I would recommend that you should keep your basis people in loop so that, they can monitor the DATABASE log file size on the PI system. If it grows larger than expected, the schedulers would not delete them until the time specified.

       

      few thousand records every 30 sec is not a bad idea, provided the system has enough memory and hard disk space for all the logs that PI writes for each message processed.

       

      VJ

      • Re: Maximum data volume that can flow through PI 7.1
        Piyush Mathur
        Currently Being Moderated

        Hi All,

        Thanks for your inputs.

        I have done a manual testing on my end and my findings are as below:

        1)     To pull 7K records from SQL database and push these to ISU via PI, it requires around 10 sec.

        2)     If we pull around 7K records in 1 SQL Query stat and schedule this stat to run every 20 sec then we would need around 100 hrs of scheduled time to fetch 12crore records.

         

        I was trying to do this testing via the "Performance Monitoring" of RWB but the help.sap content is insufficient.

        I even tried searching on SDN and found out multiple liinks but not a good one yet

         

        regards,

        Piyush

        • Re: Maximum data volume that can flow through PI 7.1
          Puneet Singhal
          Currently Being Moderated

          Hi Piyush,

          I just wanted to check if you considered Zip-Unzip option to handle such a heavy load.

           

          YOu can refer the following blogs for details:

           

          https://www.sdn.sap.com/irj/sdn/weblogs?blog=/pub/wlg/ 5899

           

          /people/aayush.dubey2/blog/2007/10/10/zip-transfer-unzip-increase-the-performance-of-your-java-abap-applications

           

          /people/pooja.pandey/blog/2005/10/17/number-formatting-to-handle-large-numbers

           

          /people/alessandro.guarneri/blog/2007/02/21/sap-xi-acting-as-a-huge-file-mover

           

          /people/alessandro.guarneri/blog/2006/03/05/managing-bulky-flat-messages-with-sap-xi-tunneling-once-again--updated

           

          /people/udo.paltzer/blog/2008/11/03/real-life-customer-examples-for-high-volume-scenarios-with-sap-netweaver-process-integration

           

           

          Rgds,

          Puneet

        • Re: Maximum data volume that can flow through PI 7.1
          Puneet Singhal
          Currently Being Moderated

          Piyush,

           

          Can you ensure that before using Performance monitoring, you have done the following configurations:

           

          Go to Integration Engine Configuration:

           

          1. Transaction SXMB_ADMIN

          2. Select "Integration Engine Configuration"

          3. In the "Specific Configuration Data" window, enter category "PERF" (it will appear in the drop-down list). NOTE: Do not click the button labeled "Specific Configuration" (yet).

          4. With "PERF" appearing in the text box, click the blue "i" (Info) button to the right of the text box.

          5. Scroll all the way down. The job instructions are there. The programs are RSXMS_COPY_PERF_DATA and RSXMS_PERF_REORG, and SAP provides guidance on how to schedule them.

           

          The job names SAP_XMB_PERF_COPY and SAP_XMB_PERF_REORG are SAP's recommended names for those jobs -- you actually have to type those names in; they don't exist.

           

          Note that the same help page tells you what PERF config settings you have to configure. You set those by clicking the "Specific Configuration" button on that screen, and entering the configuration on the next screen.

           

          Hpe it helps.

           

          Rgds,

          Puneet

  • Re: Maximum data volume that can flow through PI 7.1
    SUPARNA RAYCHAUDHURI
    Currently Being Moderated

    Hi Piyush,

     

    The performance would depend on the load of the PI system..You can however monitor the performance by various means.

     

    'Performance Monitoring' in RWB takes care of the Pipeline processing time and is exactly the same as the time shown on SXMB_MONI.

     

    The End-to-End Monitoring gives the AE processing time with the IE processing time but unfortunately it does not give the time for pulling the records from the databse.

    Here the AE processing time starts from 'Executed query successfully'. We get the total time for the adapter processing in Communication Channel monitoring.

     

    Hence ideally it is a summation of CC Monitoring (SenderReceiver) Performance Monitoring (either in MONI or RWB).

     

    Hope this helps..

Actions