How to pick up files with filename dynamically specified in message content

[engswee]

Update 8 Jan 2015: Add Sync-Async bridge as a use case

Introduction

Typically, file based interfaces involve pickup of files for further processing. SAP's file based adapters (NFS, FTP, SFTP) provide the file name scheme/pattern to specify which files should be picked. Although placeholders, regex and wildcards are allowed, it is still a static definition.

Occasionally, there are requirements to dynamically pick up files based on the file name provided in the content of a message payload. This cannot be achieved with the standard adapters because:

• Sender channels can only pick up based on static definitions
• Receiver channels can only deliver files but not pick files

One approach to achieve this is to code a file IO operation in a mapping UDF. Although it is technically feasible, it violates EJB 2.1 specification. SAP's PI JavaDocs clearly documents this for the package com.sap.aii.mappingtool.tf7.rt (used for Function Libraries and UDFs.)

Note for Java mapping programs (user-defined functions and Imported Archives), the programming restrictions imposed on J2EE application providers and especially the restrictions imposed on J2EE EJB providers are valid. This means for example, you should not

• load and use JDBC drivers (use instead the Lookup API),


• use class loaders,


• use the package java.io (writing to or reading from file system),


• create and manage threads,


• change system properties (using java.lang.System.setProperties()),


• listen on, accept connections on, or multicast from a network socket.

For more on these restrictions, see chapter 25.1.2 of the EJB 2.1 specification.

An example of such violation is if the file IO operation moves the file after pickup. This mapping program is now stateful and introduces a side effect. Due to the side effect, failed messages cannot be reprocessed with a guarantee that the behaviour is still the same. The following blog describes in more detail why XI mappings should be free of side effects.

Approach

This blog details an approach to achieve such requirement using the Additional Files feature of the sender File (only NFS protocol) or SFTP adapter.

The basic idea is to have two interface scenarios as described below:

• Scenario 1 - based on dynamically specified target filename to be picked up, create a trigger file in same directory
• Scenario 2 - polls the directory for the trigger file, once trigger file is found, pick up additionally the target file

Configuration

Below are the design and configuration for the scenarios

Scenario 1

Based on incoming message containing filename to be picked up, create a trigger file in the directory with the same filename except having a different extension, i.e <filename>.trigger. The actual content of the trigger file does not matter. Graphical message mapping is used to change the filename and store it in a field of the target message. Variable substitution in receiver channel is configured so that the file is created based on the trigger filename in the target payload.

Message Mapping

Replaces source extension with trigger extension in filename to be picked up

Receiver channel configuration

Filename is configured to be retrieved from payload content via variable substitution

iFlow

iFlow setup with Operation Mapping specifying the change of extension

Scenario 2

Sender adapter will pickup the trigger file and adds the target file as an additional attachment. PayloadSwapBean is used to swap the attachment into the main document. Mapping is performed to change the extension of the main document since Dynamic Configuration will have the trigger file as its filename. The target file is then delivered to its final destination.

Sender channel configuration

Channel set to pick up trigger files along with the corresponding CSV files. ASMA is set for filename

PayloadSwapBean in configured in the module chain to swap the CSV attachment into the main payload

Message/Java Mapping

Mapping is performed to change back the filename extension from trigger filename to the original filename.

For non-XML target files, use Java mapping with Dynamic Configuration to change the filename extension

Receiver Channel Configuration

Receiver channel is set to create target file based on ASMA value

iFlow

iFlow setup with Operation Mapping specifying the change of extension back

Testing Example

Target CSV file that is to be picked up from the source directory

A message is sent on scenario 1 to specify pickup of the CSV file. Mapping changes the filename extension to the trigger file

Trigger file is created in the directory with CSV file that is to be picked up

Trigger file is picked up with additional CSV file swapped into main payload

The logs display that originally the trigger file is in the main payload, and the CSV file is the attachment. After the swap, the CSV file is the main payload and the trigger file is the attachment.

Finally, the CSV file is delivered to the intended final destination.

Use cases

Below are some example of use cases for such an approach:

• Pick up specific file to move to different folder
• Pick up specific file for further inbound processing
• Pick up specific file to be attached in an email
• Sync-Async bridge to indicate filename in request message and get file content in response message. Ensure that polling frequency for sender channel in scenario 2 does not exceed the timeout setting of original synchronous interface.

Reference

Below are some reference material for configuration of "Additional Files" in the sender adapter. Note that this is only available for NFS and SFTP sender adapters.

Sender File Adapter Frequently Asked Questions - Process Integration - SCN Wiki (Question 4)

Sending Additional Files from SAP Netweaver PI to Third party FTP location