Data Services and Amazon S3 - Integration?

[Former Member]

A client just threw a Data Services question at me that I can't answer - how do you get Data Services to integrate with Amazon S3?  They want to read files from S3 into Data Services for analysis and transformation, and then write their output back to S3 (ideally).

I did some googling and surprisingly, I can't find any discussions of anyone having done this before.

Could Data Services could call FTP to move files in and out of S3?

Is there some other interface or API that could be called from Data Services using the SQL transform (via a custom stored procedure)?

Has anyone already solved this?

I would be delighted to hear your ideas or war stories.

Cheers!

Michael

[Former Member]

Hello,

DS 4.2. SP7 has added connectivity to AWS S3. You can use the "File Location Object" to extract and load data to S3.

Here is a video of how to use File Location Object"

SAP Data Services File Location Object - YouTube

Please note that in DS 4.2.5, the File Location Object only supports FTP, SFTP, SCT protocols. In DS 4.2.7, we added support for AWS S3, Google Cloud Storage, and Azure Blob Storage.

Hope this help,
Lofan

[Former Member]

Hello All,

Did anyone succeed with this integration, I really appreciate if you could share your valuable inputs.

I've to do a similar implementation as mentioned in the original post.

[former_member403017]

I have been researching the topic and using the exec function with AWS Command Line Interface might do the trick.

Didn't put it in practice yet though. Will update if I get the chance to try it.

[ankit_sharma88]

Hello Experts,

Has someone been able to crack this mystery.

I was looking for options in Data Services to insert data into S3. Extract data from an SQL Server , load the data into a staging if required and then finally insert these records into S3. What would be the best approach, moving ahead.

Has someone been able to integrate S3 and BODS??

Cheers,

[Former Member]

I have used a software called TNTDrive to make the S3 bucket look like a windows drive then integrated from there.
I was looking for a better way.

if anyone has more ideas, let me know.

Kimberly Jaeger

Solution Architect

SAP NS2

[Former Member]

Is this topic really a Bump?

Nobody on SCN has any experience, or even ideas on how Data Services integrates with S3?

[Former Member]

Have not needed to do this yet, but this did come up as a dicussion recently with a customer. Pretty sure you'll need to use a web service data store and use REST API (Data Services 14.2.2 would be required I believe for REST) as according to S3 current documentation SOAP support over HTTP has be depricated. Looks like SOAP still works over HTTPS but will not be supported with new functionality moving forward.

Looks like you create buckets on S3 and configure who and how you can access, but comunication is via web services.

Here is information for writing to S3 using buckets:

Uploading files to Amazon S3 - Amazon Redshift

Also, there are a ton of SDK's available for data exchange. Here are the docs for Python that could be used from a user defined transform in Data Services:

boto: A Python interface to Amazon Web Services — boto v2.32.0

To start I would think you would just need to create a web service data store and be familar with calling functions from DS look like.

Hope this helps.

[former_member198401]

Hi Michael,

Amazon Redshift uses industry standard SQL and is accessed using standard JDBC, or ODBC, drivers.

I have never worked with redhsift driver but you can use the below thread to read data from AWS. Hope it is useful

Data Services with Amazon S3/Redshift as target | SCN

Additional Link

Configure an ODBC Connection - Amazon Redshift

Regards

Arun Sasi