I spent a lot of time in searching if this was actually possible, did a lot of research, tried many different methods and finally I’ve been successful.
Before I start let me give you some background;
Business Scenario
The Sender system is IBM WebSphere MQ and the receiver side is an SFTP.
So it becomes JMS to SFTP.
Business Requirement
The sender side is splitting each xml document to 3 mb parts. But it is splitted irregularly.
For example;
This is the structure before they split and what receiver system expects from PI.
The xml document above becomes like below parts when client put the files to MQ.
Another requirement is, the message parts are uploaded to MQ unordered. So last message part can be at first row or first message part can be at middle rows etc.
And total size of the complete/unsplitted xml document can be max. 50+ MB.
Also quantity of message parts is unknown and most important thing is there is nothing to use to define the message parts sequence.
Solution Tries
Firstly it occurred to me that I can send message parts to ccBPM and merge them with a transformation step. Then I tried this scenario.
Because we don’t have any possible field to use in correlation, I made dummy correlation like giving ‘sender component’.
When I run this scenario, I noticed that incoming messages are failing on transformation step. I thought that reason of this exception is while messages goes in to mapping, they are tried to parse as if the message is xml.
By the way sender and receiver adapter validations are off.
To achieve it I tried to add xml elements like <Envelope> and </Envelope> at beginning of the message and end of the message to convert my irregular xml message to smooth and parsable xml message.
First I tried this method using XSLT by embedding incoming message to smooth xml structure as CDATA. This was 1. step on Operation Mapping and the 2. step was my actual mapping which I merge incoming messages as below.
This attempt failed. Because XSLT expects incoming data as xml format.
Then I used Java Mapping instead of XSLT with the same logic; embedding incoming data in to xml structure as CDATA. But this time I placed java mapping in front of the ccBPM instead of transformation step like below. This attempt failed too.
I realized that it mighty have impossible to handle irregular xml after message go out from sender channel.
Then I decided to write a simple custom module on JMS adapter to convert incoming data to xml message by adding xml tags beginning and end of the message.
After the start of writing EJB, an idea came; why I didn’t just make a runnable jar and call it from channel?
Solution
I made some changes on configuration and made it JMS to File as shown below.
PI is taking message parts from MQ and putting them in a folder on itself without any conversion.
While leaving the messages to folder, there is an operating system command on receiver file adapter which works after message processing.
With this command PI executes a script file whenever each message file created in target folder.
The script file executes a runnable jar file. But before the execution it checks whether the jar is running currently or ready to run at that moment. This check is necessary to prevent triggering jar file on every message part. It must run once when the first message part placed in the folder.
When the jar app is executed, it fetches all message part files from PI’s output folder and goes in a loop which turns number of file times.
First looks for the begin of root element if the message part contains it then it is assigned to a string. Then looks for the end of root element likewise and it is assigned to another string. Remainders is assign to another else string by concatenating each other in order and in a row.
After each assignment, currently processing file is moved to ‘processed’ folder.
At the end of the loop, which means all message part files are processed and moved from PI’s output folder, the strings assigned above are concatenated and this generates complete xml. At the same time this strings represents three main part of the complete xml which are begin part, middle parts and end part.
Then the new complete xml file is created with containing timestamp in filename to prevent duplicate situation.
The checking logic in bat file is, if jar file name is ‘ready’ then change its name to ‘running’ and execute it. Then check the folder continuously which PI puts incoming message parts, until there is no file anymore. This means jar app fetched all message parts and done its job. Then bat file changes the jar file’s name again to ‘ready’ then the script ends.
Also I made some performance tests like merging 20 files which equals to 60 mb total. During the tests I have observed that playing with big data causes Java heap size error. To achieve it I added -Xmx parameter to bat file which sets maximum java heap size.
As a result the script in front of the jar app enables to run jar app once until all message parts processed.
Script file;
@echo off
if exist "XMLMerger_ready.jar" (
rename XMLMerger_ready.jar XMLMerger_running.jar
java -Xmx512M -jar XMLMerger_running.jar
)
:loop
if not exist "T:\in\incoming\%" (
rename XMLMerger_running.jar XMLMerger_ready.jar
exit
) else (
goto :loop
)
Jar source code;
package com.xml;
/**
* @author ridvanpolat
*
*/
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.sql.Timestamp;
public class Merge {
public static void main(String[] args) throws IOException {
// wait 5 seconds to let PI to put all message parts to the incoming folder
try {
Thread.sleep(5000);
} catch (InterruptedException ex) {
Thread.currentThread().interrupt();
}
// declarations
java.util.Date date = new java.util.Date();
String beginPart = "";
String midParts = "";
String endPart = "";
// we will put generated xml to this path
String path = "T:\\in\\";
// we will fetch message parts from this path which PI puts
File file = new File(path + "\\incoming\\");
// check whether the path is exist or not
if (file != null && file.exists()) {
// get all files to an array, not only the filename, gets with the path
File[] listOfFiles = file.listFiles();
// if there is file in the folder then..
if (listOfFiles != null) {
// we will use this string to collect all parts at the end
String completeXML = "";
// run the codes below number of file times
for (int i = 0; i < listOfFiles.length; i++) {
// this control is necessary because there might be some other non-file items in the folder, we have to process only files.
if (listOfFiles[i].isFile()) {
BufferedReader br = null;
try {
// for the current file. we are reading first line only, because the part files are composed of single line
String sCurrentLine;
br = new BufferedReader(new FileReader(listOfFiles[i]));
// if file is not empty
while ((sCurrentLine = br.readLine()) != null) {
// get begin part
if (sCurrentLine.contains("<Envelope")) {
beginPart = sCurrentLine;
// check if the message is complete xml
if (sCurrentLine.contains("</Envelope")) {
completeXML = sCurrentLine;
}
// get end part
} else if (sCurrentLine.contains("</Envelope")) {
endPart = sCurrentLine;
// get middle parts and concat
} else {
midParts = midParts.concat(sCurrentLine);
}
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (br != null) {
br.close();
}
} catch (IOException ex) {
ex.printStackTrace();
}
}
try {
// move processed files from incoming folder
if (listOfFiles[i].renameTo(
new File(
path + "\\processed\\"
+ listOfFiles[i].getName()))) {
System.out.println("File is moved successful!");
} else {
System.out.println("File is failed to move!");
}
} catch (Exception e) {
e.printStackTrace();
}
}
// endloop
}
// concat 3 main part in order and in a row
completeXML = beginPart.concat(midParts.concat(endPart));
// for error handling from script file
System.out.println(completeXML);
BufferedWriter output = null;
try {
// get time stamp
String getTimestamp = new Timestamp(date.getTime()).toString();
// change unsupported characters for filename
getTimestamp = getTimestamp.replace(' ', '_').replace(':', '-').replace('.', '-');
// create new file with xml extention and timestamp to prevent duplicate situation
File newFile = new File(path + "\\XMLdata" + getTimestamp + ".xml");
output = new BufferedWriter(new FileWriter(newFile));
// push our concated data to new file
output.write(completeXML);
} catch (IOException e) {
e.printStackTrace();
} finally {
if (output != null) {
output.close();
}
}
}
}
}
}
I hope has been useful.
Ridvan Polat
- SAP Managed Tags:
