cancel
Showing results for 
Search instead for 
Did you mean: 

Real time data replication by using HANA EIM (Smart Data Integration)

Former Member
0 Kudos

I am trying to set up real-time replication task to replicate VBRK table from SAP ECC using ECC adaptor. I followed the steps outlined by HANA Academy in this video.

https://www.youtube.com/watch?v=eCgSB0y7_hI&list=PLkzo92owKnVwQ_preA3cxlQjn_v3W0Eh5&index=6

When I call procedure for this replication, I get the following error.

Could not execute 'call "SAPD02"."TEST_SDI.SDQ_SDI::REAL_TIME_REP_SP"' in 10:12.921 minutes .

[129]: transaction rolled back by an internal error: [129] "SAPD02"."TEST_SDI.SDQ_SDI::REAL_TIME_REP_SP": line 5 col 1 (at pos 115): [129] (range 3): transaction rolled back by an internal error: sql processing error: QUEUE: TEST_SDI.SDQ_SDI::REAL_TIME_REP_RS: Failed to add subscription for remote subscription TEST_SDI.SDQ_SDI::REAL_TIME_REP_RS.Error: exception 151050: CDC add subscription failed: Request timed out.

: line 1 col 1 (at pos 0)

Any ideas on what I may be doing wrong?

Accepted Solutions (1)

Accepted Solutions (1)

0 Kudos

Hi Abhi,

It could be many things, let's try and figure what the problem is.

Did you configure the underlying database as defined in the HANA EIM Administration Guide? 

If so, can you have a look at the following logs to see if there are any errors and post some info about it?

dp agent - /usr/sap/dataprovagent/logs/framework.log

replication admin logs - /usr/sap/dataprovagent/LogReader/admin_logs/adminxyz.log

replication instance log - /usr/sap/dataprovagent/LogReader/<instance_name>/log/<instance_name>.log

Best Regards,

Richard

0 Kudos


Hey Richard,

I'm seeing an error in the replication instance log of table rasysparms not found, and then many following subsequent errors. The other two logs don't show any errors. I've attached the log file. Thanks!

-Noah

0 Kudos

Hi Noah,

Did you perform the steps described in section 6..6.1.2 of the HANA EIM Administration Guide to configure your SQL Server?

Best Regards,

Richard

0 Kudos

Hey Richard,

Yes the 6.6.1.2 MSSQL configuration was completed in full during the initial setup.

Thanks!

Noah

Answers (2)

Answers (2)

Former Member
0 Kudos

Hi Abhishek,

We had the same error on a OracleLogReaderAdapter after we had recreated the remote source. Basically, I dropped the agent and re-added it.

If you have done this (or you need to do this), you will need to complete the following steps:

Unregister the agent. This will cause a cascade delete of the remote source and all virtual tables created.

Stop the Agent service.

In the <agent install folder> (dataprovagent), under the LogReader directory, delete the complete directory relating to the remote source attached to the agent. For example, if the remote source was attached to an instance name 'abcd' so I deleted the whole directory named 'abcd'.

In the same LogReader  directory, in the scripts directory is a *_logreader_script pertaining to your database type.

Then go through the steps to create the register the agent, register the adapter and create the remote source again.

I know it sounds painful, but it was the only way we could get it to work again.

It may pay to try the same thing for other CDC adapter types.

Hope this helps.

Jamie

Former Member
0 Kudos

We're having the same issue with NullPointerException. It seems the agent can't handle any failure during config, and if you get any failures you need to wipe all the config and start again.

We get various errors but usually starting with the below (ticket open with SAP for this)

NullPointerException while processing request "CDC Request for OracleLogReaderAdapter for request type MDS_OPEN"

Hopefully the Dev team can improve this in the next release.

Former Member
0 Kudos

Hi Jeff,

We've been working really closely with the development team on this issue and we've got it running pretty stable at the moment, but we certainly aren't running production loads through it yet.

We're currently waiting for some fixes to come in Maintenance Release 102.04 in January to see if it sorts it out some more. We're currently running on Maintenance Release 102.03 which has helped resolve a lot of the problems to date.

There were a bunch of steps we had to follow which I'm not sure if they would be appropriate for your instance, but I'm more than happy to share then if you're insterested.

The key thing for us now is to develop and activate the flowgraphs from the Web IDE but only execute them from HANA Studio by calling the Stored Procedure created.

Jamie

Former Member
0 Kudos

Thanks Jamie,

We have had some engagement with the dev guys but with it being Xmas there haven't been a lot of people around. It looks like we've got it working now though, so we'll see if it stays going once we start putting some workload through the Oracle DB.

Will keep an eye out for 102.04, but as we're just doing a PoC we could also try SPS11 if there's anything new in that, so far just seems to be some (admittedly quite nice) UI changes for the admin screens.

Cheers,

Jeff

Former Member
0 Kudos

No problems Jeff.

A lot of the fixes we've been looking for are sorted in SPS11, particularly the one we are waiting for is revision 102.04. Try and do real time replication on a table in Oracle with a column defined as NUMBER with no precision - it'll throw data mismatch errors and the workaround for it is not pretty.

Have fuin and if you could share your experience with SPS11 if you go that way it would be great.

Thanks,

Jamie

Former Member
0 Kudos

Hi Jamie,

Turns out we're still on 102.00 so next step is to get onto 102.03. We did some successful real time replication today but apply failed with an index out of bounds error which we're hoping will be fixed in 102.03. We're holding on SPS11 for now pending the results of 102.03.

Will watch out for the data mismatches, thanks for the tip.

Will update where we get to on here,

Have a great New Year,

Cheers,

Jeff

Former Member
0 Kudos

Hi,

I was wondering if a resolution to this has been found as I am experiencing the same (or a similar issue). As a summary, we are running HANA SPS10 and my remote source is SQL Server 2012. I have created a flowgraph model and can successfully execute a non-realtime flowgraph. Presumably this confirms that all the necessary prereqs (particularly on the remote database) are in place. When I select the "Realtime" option on the canvas and the data source node, I receive the following error:

SAP DBTech JDBC: [256]: sql processing error: "HANA_UK"."afm_project::REALTIME_SP": line 6 col 1 (at pos 187): [256] (range 3) sql processing error exception: sql processing error: QUEUE: afm_project::REALTIME_RS: Failed to add subscription for remote subscription afm_project::REALTIME_RS.Error: exception 151050: CDC add subscription failed: NullPointerException while processing request "CDC Request for MssqlLogReaderAdapter for request type CDC_ADD_SUBSCRIPTION"

Any advice would be greatly received.

Thanks

0 Kudos

Hi Asad,

We ended up having to increase the messagetimeout parameter under the dpserver.ini config which resolved our issue. Also, running the sql init script had to be done through sqlcmd in the command line and not through the sql mgmt studio as objects in the database didn't create properly.

-Noah

Former Member
0 Kudos

Hi Noah,

Thanks for the prompt response.

I don't see the setting you have mentioned above. I am running SPS10 and here's my dpserver.ini node:

I've also done a search for "messagetimeout" and don't find anything. The closest I can see is "default_read_timeout".

Thanks again.

0 Kudos

Hey Asad,

That framework container and messagetimeout value we had to manually create by right clicking on the dpserver.ini and selecting add section. Make sure to enter it exactly as it's displayed in the previous screenshot. We received the parameter and value for the parameter from the dpserver development team at SAP. Originally we had it set to 3600 seconds and increased it to it's current value of 21600 seconds to ensure all large tables wouldn't timeout. Without adding that section the dpserver defaults to 10 minutes as I recall and we would receive generic error messages in the hana studio. There might be additional details under the logreader folder either in the framework logs or in the instance log under the logreader folder if you're having the same problem as we were. Hopefully this works for you!

-Noah

Former Member
0 Kudos

Hi Noah,

Thanks for the clarification

Would love to report it's fixed, but...

Let me start by confirming that I've created the dpserver.ini property and set the values as you have suggested. I then used the DAC to run the mssql_server_init.sql via SQLCMD. Both of these operations appeared to complete successfully.

When I now run the realtime job, I get the following error:

Could not execute 'CALL "ITELLI_UK"."Asad_XSProject::DataRep_SP"()' in 352 ms 591 µs .

SAP DBTech JDBC: [256]: sql processing error: "MY_USER"."Asad_XSProject::DataRep_SP": line 6 col 1 (at pos 191): [256] (range 3) sql processing error exception: sql processing error: QUEUE: Asad_XSProject::DataRep_RS: Failed to add subscription for remote subscription Asad_XSProject::DataRep_RS.Error: exception 151050: CDC add subscription failed: NullPointerException while processing request "CDC Request for MssqlLogReaderAdapter for request type CDC_ADD_SUBSCRIPTION"

I've done some further digging:

- The EIM manual suggests that the init script is not required when the remote source is using CDC. My remote source is configured with CDC. I ahve also switched this to "native". It returns the same error

- The framework.trc contains the following:

2015-08-09 12:29:31,466 [INFO ] com.sap.hana.dp.mssqllogreaderadapter.MssqlLogReaderAdapter.addSubscription[267]    - [LONSDIS01.ITELLIGENCE.ORG]Subscribe a table ["dbo"."Person"] with specification [SubscriptionSpecification [header=remoteTableId=55, remoteTriggerId=2273668, sql=SELECT "T1"."ProductID", "T1"."Name", "T1"."ProductNumber", "T1"."MakeFlag", "T1"."FinishedGoodsFlag", "T1"."Color", "T1"."SafetyStockLevel", "T1"."ReorderPoint", "T1"."StandardCost", "T1"."ListPrice", "T1"."Size", "T1"."SizeUnitMeasureCode", "T1"."WeightUnitMeasureCode", "T1"."Weight", "T1"."DaysToManufacture", "T1"."ProductLine", "T1"."Class", "T1"."Style", "T1"."ProductSubcategoryID", "T1"."ProductModelID", "T1"."SellStartDate", "T1"."SellEndDate", "T1"."DiscontinuedDate", "T1"."rowguid", "T1"."ModifiedDate" FROM """dbo"".""Person""" "T1" , subscription=, customId=, seqID=SequenceId [value=[0, 0, 0, 0]]Last Subscription=true : withSchemaChanges=false]List of Columns: INTEGER         NULL           

                                NVARCHAR        NULL           

                                NVARCHAR        NULL           

                                TINYINT         NULL           

                                TINYINT         NULL           

                                NVARCHAR        NULL           

                                SMALLINT        NULL           

                                SMALLINT        NULL           

                                DECIMAL (19,4)  NULL           

                                DECIMAL (19,4)  NULL           

                                NVARCHAR        NULL           

                                NVARCHAR        NULL           

                                NVARCHAR        NULL           

                                DECIMAL (8,2)   NULL           

                                INTEGER         NULL           

                                NVARCHAR        NULL           

                                NVARCHAR        NULL        ...

2015-08-09 12:29:31,466 [ERROR] com.sap.hana.dp.adapter.framework.core.WorkerThread.processRequest[241]    - NullPointerException while processing request "CDC Request for MssqlLogReaderAdapter for request type CDC_ADD_SUBSCRIPTION" Context: java.lang.NullPointerException

                                at com.sap.hana.dp.mssqllogreaderadapter.MssqlLogReaderAdapter.addSubscription(MssqlLogReaderAdapter.java:269)

                                at com.sap.hana.dp.adapter.framework.core.WorkerThread.processAddSubscription(WorkerThread.java:545)

                                at com.sap.hana.dp.adapter.framework.core.WorkerThread.processRequest(WorkerThread.java:179)

                                at com.sap.hana.dp.adapter.framework.core.WorkerThread.run(WorkerThread.java:1008)

                                at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

                                at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

                                at java.lang.Thread.run(Unknown Source)

                                com.sap.hana.dp.adapter.sdk.AdapterException: java.lang.NullPointerException

                                at com.sap.hana.dp.adapter.framework.core.WorkerThread.processRequest(WorkerThread.java:240)

                                at com.sap.hana.dp.adapter.framework.core.WorkerThread.run(WorkerThread.java:1008)

                                at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

                                at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

                                at java.lang.Thread.run(Unknown Source)

                                Caused by: java.lang.NullPointerException

                                at com.sap.hana.dp.mssqllogreaderadapter.MssqlLogReaderAdapter.addSubscription(MssqlLogReaderAdapter.java:269)

                                at com.sap.hana.dp.adapter.framework.core.WorkerThread.processAddSubscription(WorkerThread.java:545)

                                at com.sap.hana.dp.adapter.framework.core.WorkerThread.processRequest(WorkerThread.java:179)

                                ... 4 more

I've un-installed and re-installed Java SDK on the server but this hasn't helped. A weekend in and still no realtime job

Thanks for any help anyone can offer.

0 Kudos

Hi Asad,

The MssqlLogReaderAdapter doesn't use the SQL Server CDC functionality, it reads the transaction log directly to get changes in real time.  In order for this to work, you must run the initialization script.

Can you please post the contents of the instance log (found in /usr/sap/dataprovagent/LogReader/<instance_name>/log).

This log file usually contains more information to help troubleshoot the problem.

Best Regards,

Richard

Former Member
0 Kudos

Hi Richard,

Thanks you for the response and the suggestions you provided.

Let me share the results with you.

After switching the “Database Data Capture Mode” property on the “Remote Source” to “Native Mode” and re-executing the flowgraph, I received the following error:

Could not execute 'CALL "MYJOB_UK"."Asad_XSProject::DataRep_SP"()' in 509 ms 885 µs .

SAP DBTech JDBC: [256]: sql processing error: "MYJOB_UK"."Asad_XSProject::DataRep_SP": line 6 col 1 (at pos 191): [256] (range 3) sql processing error exception: sql processing error: QUEUE: Asad_XSProject::DataRep_RS: Failed to add subscription for remote subscription Asad_XSProject::DataRep_RS.Error: exception 151050: CDC add subscription failed: No original/mirror paths mapping found for database <VITA> in file <C:\usr\sap\dataprovagent\LogReader\config\mssql_log_path_mapping.props>.

Upon examining the mssql_log_path_mapping.props file, I think this is only required when the Agent is installed on a server other than the source database. My source (remote db) and the agent are installed on the same server. This led me back to the configuration of the remote source. I then switched the “Use Remote Database” to “false”.

Upon making this change and re-executing the flowgraph, I now receive the following error:

Could not execute 'CALL "MYJOB_UK"."Asad_XSProject::DataRep_SP"()' in 868 ms 335 µs .

SAP DBTech JDBC: [256]: sql processing error: " MYJOB _UK"."Asad_XSProject::DataRep_SP": line 6 col 1 (at pos 191): [256] (range 3) sql processing error exception: sql processing error: QUEUE: Asad_XSProject::DataRep_RS: Failed to add subscription for remote subscription Asad_XSProject::DataRep_RS.Error: exception 151050: CDC add subscription failed: NullPointerException while processing request "CDC Request for MssqlLogReaderAdapter for request type CDC_ADD_SUBSCRIPTION"

: line 1 col 1 (at pos 0)

Please note that the CDC property has already been switched as per your suggestion. For clarity, here’s a screenshot of my remote source:

I have also been checking the log files you have mentioned and they only contain repeated entries of the following:

[Fri Aug 14 10:30:45 2015 ] Receive Ping reply message from DP Agent.

[Fri Aug 14 10:30:55 2015 ] Sent a Ping message to DP Agent.

I had already executed the mssql_server_init.sql script. I did this using the DAC via SQLCMD and it reported success. However, I don’t see any of the SP’s listed in the System Procs of the individual databases. This surprised me as I was expecting to see Procs such as “sys.sp_SybSetLogforReplTable” in all of the databases. Should that be the case?

Thanks again.

Asad

Former Member
0 Kudos

Hi Asad,

did you get the solution of this issue?

Can you please help If you got the solution, I am also facing the same issue.

Thanks,

Amit

Former Member
0 Kudos

Hi Amit,

Nothing as yet. I've had a call with SAP which dates back over a month now. I am still waiting on a resolution. I will post it on here once it's available.

Are you encountering the same symptoms? Have you tried a source other than SQL Server? I have it working with SAP HANA and Twitter.

Regards,

Asad

Former Member
0 Kudos

Hi Noah ,

I have integrated MSSQL server with SAP HANA through SDI mssqllogreader Adapter.I am trying to preplicate  VBAK table into HANA.

I faced the similar issue . The error is  ] (dberror) 2048 - column store error: [2048] "schema"."Taskname": line 5 col 0 (at pos 415): [2048] (range 3): column store error: task framework: [403] internal error: Remote execution error Prefetch timed out. for query .

I tried the option by increasing the DP server Message timeout  to 21600.

But still i am facing the issue.

Kindly suggest me how should i resolve the issue.

Regards,

Sidharth

0 Kudos

Hi Sidharth,

Please try the following.

ALTER SYSTEM ALTER CONFIGURATION ('dpserver.ini','SYSTEM') SET ('framework', 'prefetchTimeout') = '9999'  WITH RECONFIGURE;

Best Regards,

Richard