SAP BODS to load into HADOOP/HIVE

[former_member188035]

Hi Experts,

Please help me the capabilities of SAP BODS as an ETL tool to load data (RDBMS data both master and transaction) into HADOOP/HIVE. Any pointers reg specific transformations with native MapReduce/Yarn support will be of great help.

[former_member187605]

The DS HIVE adapter is read-only. Your only option to load data into Hadoop is thru an HDFS file format. You use an HDFS file as target in a data flow as you would use any other flat file format;

Up till DS 4.2 SP1, Hadoop 1.0 is supported. DS 4.2 SP2 supports Hadoop 2.0 only.

[former_member188035]

Thanks Dirk, But if I am going to realize my data as flat files in HADOOP then how much possible its to do a Update/Append operation over it? Because as I mentioned earlier its going to be RDBMS data with incremental load. Does BODS offers any support for this or I have to write custom scripts/MR for handling the same ?

[former_member187605]

Append is no issue in Hadoop 2.0, fully supported.

IMO update is never possible. You'll have to create a completely new file with full contents at every incremental job run..

[former_member188035]

Dirk, Thanks again. If we load the data directly to HDFS as files, then would I be able to do a SQL like operations over the it ? in other words will HIVE be able to recognize those file-data as tables?

[former_member187605]

Yes, that's the whole idea behind HIVE. It's not more than a virtual relational layer built on top of the Hadoop file system.In fact, every HiveSQL command is implicitly converted into a MapReduce job.