Data Services - where does the data reside ?

[former_member183909]

I'm new to data services.  We are considering using it for Data Migration.

I understand that to migrate source tables to a new system in data services you build the Source table and the Target table in the Data Services Repository.

Wondering about the process to get data from Source system to the Target system.

Assuming you have built the Source table and the Target table in the BODS Repository and you can ODBC connect to extract the data.

- Is this data stored in the BODS Source table ?

- Is this data stored in the BODS Target table ?

- If you need to produce a Material Master IDOC will you need to have all of the related SAP tables set up as Targets in BODS Repository and map the Source table to all the related SAP tables.  Then build the IDOC from table data collected in MARA, MARC, MVKE etc

Thanks

[former_member183909 · Accepted solution]

Hi - thanks for the replies.

- wow !   I am surprised that BODS does not record data records. 

So extracted source data records and transformed target data is not stored in BODS.  So the repository database does not need to be sized according to source database  e.g. if source database in 8TB then I would NOT need similar size in BODS repository.

So if there is a correction made in the target transformation I would need to re-extract the source data again ? 

Also if I am going cleansing on source data in BODS, adding new cleansing rules and applying them would require re-extract of source data again.  

[former_member200473 · Accepted solution]

Jonny,

You are correct

Shiva

[Former Member · Accepted solution]

Hello

As mentioned above, the Data Services' repository stores metadata only (and some data relating to execution statistics, etc.), however most ETL projects also use a staging database and this is where the intermediate results of the processing would be stored. 

So in your example (and in most simple projects) once Data Services is configured and working, a developer would create 3 datastores (or file formats, etc) - source, staging and target.  Jobs would read from the source into the staging database, transformation occurs in the staging area, and once complete, loaded into the target.  The choice of staging location is made by the developer, it should be HANA , but it could be Oracle, SQL Server, DB2, text files, XML - although a DBMS will give significant advantages.

This means that Data Services does not store data, but your ETL project does store data.  I don't think I've ever worked on an ETL project that didn't use a staging area.

Michael

[Former Member · Accepted solution]

Hi,

On a side note: Don't underestimate the repository database, as it can grow big, so give careful consideration when sizing it as well. In a normal circumstance plan one local repository database per developer and a central repo for all.

Nawfal Tazi

[kamal264 · Accepted solution]

Also if I am going cleansing on source data in BODS, adding new cleansing rules and applying them would require re-extract of source data again. 

If you are doing cleansing in BODS then no need to re extract the data.

~Kamal

[Former Member]

Hello Jonny,

Data services (Data Integrator + Data Quality) is tool used to create batch jobs and real time jobs for ETL and data quality purpose and we can perform some data profiling too. The Job servers are responsible in handling the jobs which are managed by Job server Manager. Data services never holds any data its just a medium which allows to maintain Source and target systems meta data. Yes Data services does hold meta data related to source and target systems in a repository. In Data services just the Connections(Meta data) to the source and target are defined and maintained in Data services designer, depending on the kind of source and target systems it is going to vary the kind of connection you have to make use. For ETL and Data Quality, Data services has in built with predefined transforms(for transformation). If the data transformation you are looking out for cannot be achieved with existing transforms you can make use of script objects and can script own transforms depending on the kind of source and target systems your are making use of. Please find the architecture of the Data services below.

Best Regards,

Yusuf.

[Former Member]

If you are intending to migrate data into SAP via IDocs, you should consider using the FREE best practice data migration content provided by SAP (and already mentioned).  The material can be found here - https://websmp103.sap-ag.de/rds-dm2erpcrm.

It also uses a staging area .

Michael

[Former Member]

Jonny,

What are you migrating from and to?

As others have said the repository is for projects jobs metadata.

Sounds like you are migrating to SAP.

you could look at the "SAP Best Practices for Data Migration V1.40" stuff

build your extracts from the legacy system(s) and hook to the template tables in BP Idoc jobs.

It covers a lot of ground. has validation and generally works.

jmho

Jim

[kamal264]

Hi Jonny,

As mentioned in above comments

We create different Databases instead of Repository and connect that database in data services as Data store.

SAP Tables:- No need of database for MARA, MARC etc just create a connection with SAP in BODS means SAP Datastore  and apply the whatever logic you want.

~Kamal

[Former Member]

Hi Johnny,

Note that we never create tables or store external data in BODS repository.

BODS repository is only used to store the metadata only, for instance the job definition,  databases connection information, etc.

To migrate source tables to a new system, we can make the  source and target databases connection in DS designer, obtain the corresponding table definitions, and then create a new job to migrate the data from tables in source DB to tables in new DB.

Data resides in source DB, in-memory or file caching when processing, and then target DB. They are not stored in BODS repository.

Please find the DS tutorial guide which is written for DS beginners:

http://help.sap.com/businessobject/product_guides/sboDS41/en/sbo411_ds_tutorial_en.pdf

For data migration, there is a new feature in DS called "Workbench". You can find the Workbench guide here:

http://help.sap.com/businessobject/product_guides/sboDS41/en/sbo411_ds_workbench_en.pdf

Regards,

Gordon

[former_member200473]

Hi Jonny,

Data will never be stored into BODS.When ever you create the table in BODS,it gets created to the database.Through BODS you just create the structure for the target table and that structure also gets saved into Database.

(AS per My knowledge)As we know that there are two types of Tables in BODS (Template and Permanent tables),the difference here in both tables is in permanent table Schema gets imported to BODS repository and in Template table the schema will always be in Database.

As far as my knowledge says,you can never create source table,you can import it from different database system or ERP system.

Regards,

Shiva Sahu