SAP Data Services and SAP Replication Server

[david_pugh]

SAP Data Services & SAP Replication Server

SAP Data Services had the ability in previous versions to use SAP Replication Server for Change Data Capture capabilities but this involved the use of PowerDesigner and a staging area using Sybase ASE. As of Data Services 4.2 SP3 you no longer need PowerDesigner or the staging area as the integration is now much simpler and SAP Data Services can now connect directly to SAP Replication Server.

SAP Data Services connects to the source systems to collect metadata and then to SAP Replication Server / Agent for configuration and change data retrieval.

You create 2 datastores, one with No CDC that will be used for the initial load

and one using the CDC option which will be used for the delta loads.

Under the advanced section you now add the Replication Server and Replication Agent details.

(This example is using Sybase ASE as the source system, as of Data Services 4.2 SP3 Sybase ASE is not supported as a CDC source. Please refer to the Product Availability Matrix for supported systems)

Import the same tables from both datastores.

You can now start creating your job. You may want to start with a conditional workflow to check if the job is needs to do an initial or delta load. The initial load is just a standard dataflow using the table from the Initial datastore.

The dataflow for the delta (Rep Server CDC) part of the job has to be inside a Continuous Workflow.

Once the job has been started it will continue to run and fetch changed data from the Replication Server. The Continuous Workflow has options (e.g. number of runs or custom function) to control how long it should run for.

The dataflow inside the continuous workflow now reads the table from the Delta datastore.

When viewing the data from the Delta datastore you will notice 2 additional columns, DI_SEQUENCE_NUMBER and DI_OPERATION_TYPE. Operation type is used to determine if the record is an insert, update or delete record.

In the example dataflow above the Operation type is then used in a Map_CDC_Operation transform to tell SAP Data Services to generate update, insert or delete statements. If you don’t want to delete data from the target then a Map_Operation transform can be used to turn a delete into an update for example.

With SAP Data Services 4.2 SP3 this new simplified architecture makes it even easier to perform real time transformations as part of a data integration process.