Records are repeating in every data package of PSA

[Former Member]

Hello Experts,

I have created generic datasource using Function module. After Init with no data transfer I am running a full load and have to fetch approx 500000 data records from source system.Now data is getting loaded in PSA in different data packets but it is repeating i.e in first data packet it is loading 18000 records then in second it is loading 43000 records including 18000 of first datapacket and so on. Due to this data load in PSA is very slow. Is their any way of bringing all these 500000 in one datapacket or different records in different datapackets. Please suggest.

[Former Member · Accepted solution]

Hello Everyone,

Any Suggestions on resolving this issue? Actually full load is running since last 3 days now, if you have ever come across such scenario then please let know how you resolved it.

[KodandaPani_KV · Accepted solution]

Hi,

i had same issue with the intial days - my data source have keep on coming the records.

source side have the only 250 records but bw side keep on coming 250x numbbe of times.

the info pacakge not yet compelted keep on running.

after i check with the my ABAPer with open and fetch curser.

check the below code with abper - i think here have problem.

OPEN CURSOR WITH HOLD S_CURSOR FOR

       SELECT   banfn bnfpo ekgrp txz01 matnr matkl werks lgort menge meins badat lfdat erdat bsmng ebakz frggr

          FROM eban

          WHERE BANFN IN L_R_BANFN

          AND WERKS  IN L_R_WERKS

          AND BADAT IN L_R_BADAT

          AND ERDAT IN L_R_ERDAT

          AND pstyp = '0'

          AND loekz = ' '.

     ENDIF.                             "First data package ?

* Fetch records into interface table.

*   named E_T_'Name of extract structure'.

     FETCH NEXT CURSOR S_CURSOR

                APPENDING CORRESPONDING FIELDS

                OF TABLE gt_eban

                PACKAGE SIZE S_S_IF-MAXSIZE.

SORT GT_EBAN BY BADAT.

     IF SY-SUBRC <> 0.

       CLOSE CURSOR S_CURSOR.

       RAISE NO_MORE_DATA.

     ELSE.

IF gt_eban[] IS NOT INITIAL.

Thanks,

Phani.

[Former Member · Accepted solution]

Hi Vishal,

Can you confirm that iin each next package you are receiving all the records of previous package? It looks like if you receive 18627 records in first package and then in second package you receive 18627+ next package records. Similar like in next each package (18627+records of 2nd package+records of 3rd package) and so on.

So I still say that there is some table which is not being cleared. Can you copy paste the code where you process the records for gt_eban? I do not see much issues with the code you sent for fetch and open cursors.

Thanks

Amit

[Former Member · Accepted solution]

Hi Amit,

We need to clear gt_eban before code

S_COUNTER_DATAPAKID = S_COUNTER_DATAPAKID + 1.

It resolved my issue.

Thanks & Regards,

Vishal

[KodandaPani_KV]

Hi,

copy of the FM - RSAX_BIW_GET_DATA_SIMPLE - use the your custom FM as per the requirment.

Thanks,

Phani.

[RamanKorrapati]

Hi,

After the init never run full load. it makes duplciates into bw side.

But you can run repair full load with proper selections.

Other wise  just delete your init.

run full info pack.

run info pack - init without data transfer.

once some changes are done at source, you can trigger delta info pack.

Thanks

[Former Member]

Hi,

I have deleted Init and doing full load now but still problem persist. Duplicate records are coming, but in different datapackages . Can we do anything for bringing unique records.

[RamanKorrapati]

Hi,

Assuming as your doing at dev environment.

please check your function module code/logic with help of abap expert.

Try to debug your data source at RSA3 and check it.

Thanks

[KodandaPani_KV]

Hi,

are you the FM based generic data source if yes check the data in the RSA3 ECC

how many records is geeting in ECC

use the FM BI related - RSAX_BI_GET_DATA_SIMPLE

check the logic for the FM and open curser and close curser.

Thanks,

Phani.

[Former Member]

Hi Vishal,

It looks like your FM for the datasource have some problem. Most probably there must be some table which should be cleared after each execution and it is not being cleared. Check the code of your FM, you will get a fair idea.

If you could paste your code here then we can precisely tell where you need the change.

Thanks

Amit

[Former Member]

I have used RSAX_BIW_GET_DATA_SIMPLE FM for my generic datasource and the code logic is standard i.e i replaced existing logic with my logic for data fetching and bringing it in E_T_DATA.

Just have a look at the code and let me know if i am missing anything.

   TABLES: eban.

   DATA: L_S_SELECT TYPE SRSC_S_SELECT.

   STATICS: S_S_IF TYPE SRSC_S_IF_SIMPLE,

   S_COUNTER_DATAPAKID LIKE SY-TABIX,

   S_CURSOR TYPE CURSOR.

* Select ranges

   RANGES: L_R_banfn  FOR eban-banfn,

           L_R_werks  FOR eban-werks,

           L_R_badat  FOR eban-badat,

           L_R_erdat  FOR eban-erdat.

   DATA: LWA_DATA LIKE LINE OF E_T_DATA.

* Initialization mode (first call by SAPI) or data transfer mode

* (following calls) ?

   IF I_INITFLAG = SBIWA_C_FLAG_ON.

APPEND LINES OF I_T_SELECT TO S_S_IF-T_SELECT.

* Fill parameter buffer for data extraction calls

     S_S_IF-REQUNR    = I_REQUNR.

     S_S_IF-DSOURCE = I_DSOURCE.

     S_S_IF-MAXSIZE   = I_MAXSIZE.

* Fill field list table for an optimized select statement

* (in case that there is no 1:1 relation between InfoSource fields

* and database table fields this may be far from beeing trivial)

     APPEND LINES OF I_T_FIELDS TO S_S_IF-T_FIELDS.

   ELSE.                 "Initialization mode or data extraction ?

************************************************************************

* Data transfer: First Call      OPEN CURSOR + FETCH

*                Following Calls FETCH only

************************************************************************

* First data package -> OPEN CURSOR

     IF S_COUNTER_DATAPAKID = 0.

* Fill range tables BW will only pass down simple selection criteria

* of the type SIGN = 'I' and OPTION = 'EQ' or OPTION = 'BT'.

       LOOP AT S_S_IF-T_SELECT INTO L_S_SELECT WHERE FIELDNM = 'BANFN'.

         MOVE-CORRESPONDING L_S_SELECT TO L_R_BANFN.

         APPEND L_R_BANFN.

       ENDLOOP.

       LOOP AT S_S_IF-T_SELECT INTO L_S_SELECT WHERE FIELDNM = 'WERKS'.

         MOVE-CORRESPONDING L_S_SELECT TO L_R_WERKS.

         APPEND L_R_WERKS.

       ENDLOOP.

        LOOP AT S_S_IF-T_SELECT INTO L_S_SELECT WHERE FIELDNM = 'BADAT'.

         MOVE-CORRESPONDING L_S_SELECT TO L_R_BADAT.

         APPEND L_R_BADAT.

       ENDLOOP.

       LOOP AT S_S_IF-T_SELECT INTO L_S_SELECT WHERE FIELDNM = 'ERDAT'.

         MOVE-CORRESPONDING L_S_SELECT TO L_R_ERDAT.

          L_R_ERDAT-SIGN = 'I'.

          L_R_ERDAT-OPTION = 'GE'.

         CLEAR L_R_ERDAT-HIGH .

         APPEND L_R_ERDAT.

       ENDLOOP.

OPEN CURSOR WITH HOLD S_CURSOR FOR

       SELECT   banfn bnfpo ekgrp txz01 matnr matkl werks lgort menge meins badat lfdat erdat bsmng ebakz frggr

          FROM eban

          WHERE BANFN IN L_R_BANFN

          AND WERKS  IN L_R_WERKS

          AND BADAT IN L_R_BADAT

          AND ERDAT IN L_R_ERDAT

          AND pstyp = '0'

          AND loekz = ' '.

     ENDIF.                             "First data package ?

* Fetch records into interface table.

*   named E_T_'Name of extract structure'.

     FETCH NEXT CURSOR S_CURSOR

                APPENDING CORRESPONDING FIELDS

                OF TABLE gt_eban

                PACKAGE SIZE S_S_IF-MAXSIZE.

SORT GT_EBAN BY BADAT.

     IF SY-SUBRC <> 0.

       CLOSE CURSOR S_CURSOR.

       RAISE NO_MORE_DATA.

     ELSE.

IF gt_eban[] IS NOT INITIAL.

Here I have written further selection logic and brought the final data in E_T_DATA.

MOVE-CORRESPONDING <gs_list> TO LWA_DATA.

APPEND LWA_DATA TO E_T_DATA.

ENDLOOP.

REFRESH : gt_list[].

ENDIF.

     S_COUNTER_DATAPAKID = S_COUNTER_DATAPAKID + 1.

   ENDIF.              "Initialization mode or data extraction ?

My issue is, In RSA3 when I check data then different records are coming in each data package but when I am doing full load in BI side. It is bringing duplicate records in different datapackage as below

DP         Records

000001	18627
000002	37341
000003	56282
000004	75208
000005	93910
000006	112431

In DP2 I am getting all records of DP1 and in DP3 I am getting all records of DP2 and so on.

In total I have approx 500000 records.