How to filter and read large data in SAP table using IDM Job ?

[Former Member]

Hi,

We need to read data from SAP table which has humongous amount of data in it. When we use a standard  FromSAP pass it keep giving us lot of out of memory error. Not sure how can we include FILTER parameter here for Entry Type TABLE.

Is FILTER is applicable only at the entry type level or even for the columns ?

Experts can you please advise what could be done under this scenario ?

Thanks,

Karthik

[Former Member · Accepted solution]

Hello Experts,

Apart from Filter is there a way to set the batch size in fromSAP pass to overcome this memory issue ?

Thanks,

Karthik

[ChrisPS · Accepted solution]

Hello Karthikeyan,

                            unfortunately with the SAP connector a filter on the read Table is not implemented which is why the filter options did not work. Maybe you can use a workaround like load the table first to file and transfer to IdM and then filter it.

Regards,

Chris

SAP IdM Support

[Former Member · Accepted solution]

Thanks Chris for confirming !!

~ Krishna.

[Former Member]

Hi Karthikeyan,

When the entry type is TABLE in the FromPass, to read the specified table, the function module RFC_READ_TABLE is called.

As per my understanding, there is a known limitation called DATA_BUFFER_EXCEEDED with RFC_READ_TABLE function.

"The function RFC_READ_TABLE reads table records into a table with the structure TAB512. As the name somewhat indicates, this table has a single field with the length of 512 characters. This means you are limited to reading a combination of fields from your table that does not exceed 512 characters in width. In most cases, if you are limiting your call to the specific fields you need, this should not be a problem. However, you should be aware of this limitation. Exceeding 512 characters in width will result in an exception of DATA_BUFFER_EXCEEDED"

This, you can overcome by creating a copy of the function module, say ZRFC_READ_TABLE and changing it according to your requirement.

Not sure whether this is the thing causing error in your case, but I would ask you to double check in case this is the reason !!

I also doubt whether the filters are applicable for entry type TABLE, as I haven't found any documentation/notes related to that !!

Lets wait for someone from SAP confirms on this...

For more information on the function module, refer to

http://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/a83ec690-0201-0010-14ac-bd1d75e24...

~ Krishna

[Former Member]

Thanks all for your suggestions  🙂

As there is no filters applicable for entry type TABLE I went with function module solution.

I used a pass called 'FromSAPIdentity' with EntryType Function and could able to achieve our requirement.

For more info on this please check

(http://wiki.scn.sap.com/wiki/display/Security/Technical+Overview+of+BAPIs+used+to+connect+to+Busines...)

~ Karthik

[mabujan]

Hi Karthik,

We are facing the same challenge.

Need to read table HRP1001, which is huge, with a particular selection criteria to reduce the amount of records we need to transfer.

I am willing to go the "FUNCTION" way, but I am unsure how to enter the parameters on IDM.

Where can I read some information about this?

Would you be able to share an screenshot of your IDM setup to call a custom FUNCTION to retrieve data from SAP?

Appreciated,

Manuel

[Former Member]

Hello Manuel,

You can also try the workarounds mentioned in this post.

~ Krishna.

[Former Member]

Hi Manuel,

Below is what i configured using FromSAPIdentity pass in IdM SP7 and it worked for me

Parameter	Value
ENTRY TYPE	FUNCTION
FUNCTION	<FUNCTION MODULE NAME>
PARAMETERS	<IMPORT PARAMETER>   e.g DEPARTMENT=X
NAMEFIELD	<EXPORT PARAMETER:RETURN FIELD1 |EXPORT PARAMETER:RETURN FIELD2> e.g. EMPLOYEE:NAME|EMPLOYEE:WORKPLACE

Cheers,

Karthik

[Former Member]

Hi Karthikeyan,

I have just posted a solution which worked for me when I was in your scenario. There is a thread now open with a similar problem but a different table http://scn.sap.com/message/14621534#14621534

I hope this also works for you,

Kind regards,

Andy

[Former Member]

Hi,

Please find below the custom pass with filter included but still i am getting the same exception. It looks i haven't configured the filter properly or it is not as expected for the view we have  in SAP. Can you please advise where i am going wrong in filter configuration ?

Exception that i get in the DSE log

Unhandled Error in DSERuntime java.lang.OutOfMemoryError: Requested array size exceeds VM limit (failed to allocate 141817872 bytes) (array length 70908928)

at com.sap.mw.jco.JCO$Record.ensureBufferCapacity(JCO.java:12112)

at com.sap.mw.jco.rfc.MiddlewareRFC$Client.nativeExecute(Native Method)

at com.sap.mw.jco.rfc.MiddlewareRFC$Client.execute(MiddlewareRFC.java:1244)

We also trying to increase memory of dispatcher but it will be good if we could able to make it work with filter

Thanks,

Karthik

[Former Member]

Hi karthik,

Try without the parameters:

ENTRYTYPE

TABLE

COLUMNS

NAMEFIELD

NAMEFIELD2

NAMEFIELD3

and put the filters at the end.

Or if I'm wrong you can fry to change the filters:

FILTER         SELECTION_EXP:0/KNA1=AND

FILTER1       SELECTION_EXP:0/ARITY=3

FILTER2       SELECTION_EXP:1/PARAMETER=KTOKD

FILTER3       SELECTION_EXP:1/FIELD=GROUP

FILTER4       SELECTION_EXP:1/OPTION=CP

FILTER5       SELECTION_EXP:1/LOW=Z1*

FILTER6       SELECTION_EXP:2/PARAMETER=KTOKD

FILTER7       SELECTION_EXP:2/FIELD=NR(if you need to select by this column)

FILTER8       SELECTION_EXP:2/LOW=(if it's needed)

FILTER10     SELECTION_EXP:3/PARAMETER=KTOKD

FILTER11     SELECTION_EXP:3/FIELD=OTA(if you need to select by this column)

FILTER12     SELECTION_EXP:3/OPTION=(if it's needed)

FILTER13     SELECTION_EXP:3/LOW=(if it's needed)

and again without the parameters.

BR,

Simona

[Former Member]

Thanks Simona for the quick response. I tried both  the scenarios suggested (Without mentioned parameters) but the job failed to run.

Here are the results

Exception:

Selection criterion KNA1-KTOKD is not supported

Exception:

JCO_ERROR_FIELD_NOT_FOUND: Field KNA1 not a member of BAPIUSSEXP

Please advise if any other scenario that i can try.

Also i tried to follow

https://service.sap.com/sap/support/notes/1398976

but not able to configure the filter properly yet

Thanks,

Karthik

[Former Member]

Hi  karthikeyan,

Mu sugesstion is, if you have FromCustom pass with source tab FromSAP in your source tab after the connection parameters you can add filters like this:

as you can check the needed option from here - http://help.sap.com/abapdocu_70/en/ABENLOGEXP_STRINGS.htm

BR,

Simona

[former_member2987]

Karthik,

Please repost the screen shot.  I think I can help you, but I need to make sure I'm referring to the right scenario.

Thanks,

Matt

[Former Member]

I've not played with the filter so I can't help you there.  Perhaps just increasing the memory size for the dispatcher might help.  Create a specific dispatcher for this job with a large memory allocation and see how it goes.